acceptodds
Under review as a conference paper at ICLR 2027

Trajectory-Aware Value Guidance with a Co-Trained Critic for Offline RL

Abstract

Offline reinforcement learning (RL) faces the significant challenge of deriving optimal policies from static, often suboptimal datasets. Sequence modeling methods like Decision Transformer (DT) have shown promise, but their performance is often constrained by the quality of the best trajectories in the dataset. In contrast, value-based methods have the potential to stitch together suboptimal trajectories to achieve performance beyond that of any single trajectory. Recent work has explored integrating DT with value-based approaches to enhance trajectory stitching capabilities while avoiding distribution shift in offline RL. However, existing methods typically rely on static pre-trained Q-functions, which can become misaligned with the evolving policy during training, limiting their effectiveness. We introduce Adaptive Q-Guided Decision Transformer (AQDT), an approach that dynamically co-trains the Q-function alongside DT training, creating a mutually adaptive relationship between policy and critic. AQDT adaptively balances DT learning with value-based guidance based on trajectory quality, while ensuring that the critic remains aligned with the evolving policy. On standard D4RL benchmarks, AQDT outperforms prior DT and hybrid methods, achieving a 4.6% gain in average normalized return on locomotion tasks and demonstrating robust generalization across datasets of varying quality. The code implementation is available at :Anonymous.

Primary area:
reinforcement learning
Keywords:
Offline Reinforcement Learning, Conditional Sequence Modeling, Return Conditioned Policy

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LaR1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors show that learning the Q function alongside QCS can increase the benchmark performance by a great extent.

Critical strengths
  1. The proposed algorithm demonstrates remarkable performance on multiple D4RL benchmarks.
  2. The algorithm is simple and easy to understand.
Critical weaknesses
  1. The proposed algorithm is quite incremental in the sense that it's basically QCS + critic learning, and learning the critic with an actor regulariser is a widely used technique in the field of Offline RL.
  2. Justification for the critic learning algorithm is not provided. What's the rationale behind equation (3)? There are multiple ways to implement TD-learning. For example, one may use an n-step TD loss with a fixed n(1, 2, 3, …\dotsc). Instead of restricting the computation of the n-step return inside the sub-trajectory τ<t\tau^{<t}, one may also use samples that come after time-step tt. Ablation experiments comparing the performance of these methods is missing.
  3. Experiments on more realistic datasets such as OGBench(Park et al., 2025) would make the empirical results more impactful.

References

Park, S., Frans, K., Eysenbach, B., & Levine, S. (2025, May). Ogbench: Benchmarking offline goal-conditioned rl. In International Conference on Learning Representations (Vol. 2025, pp. 94937-94982).

Questions for authors
  1. Why is equation (1) so small?
  2. n-step returns are inherently biased. Unlike the online setting where the bias can be reduced through additional sampling, this becomes a huge problem in the offline setting. Theoretical analysis of the n-step bias and its effect on the overall algorithm would be very interesting.
Reviewer_Kn46OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies policy improvement for Decision Transformers trained on suboptimal offline data and proposes AQDT. The method adjusts the strength of Q-guidance according to the return of each complete trajectory, applying stronger value-based correction to low-return trajectories while primarily imitating high-return ones. The critic is initialized through IQL pretraining and then jointly trained with the policy using multi-step value targets. Experiments cover MuJoCo, AntMaze, and Adroit Pen tasks. The results show relatively high average returns among the compared methods, with the main gains concentrated on several suboptimal datasets.

Critical strengths
  1. The method has a clear structure and reasonable motivation. Adjusting the relative strength of imitation learning and value-based improvement according to trajectory quality helps preserve high-quality demonstrations while strengthening policy improvement on low-quality data. Section 6.5 also includes extensive factorial ablations, providing sufficient validation of the individual components.

  2. The performance improvements on several suboptimal-data tasks are substantial. For example, AQDT achieves 70.32 on HalfCheetah-medium, compared with 59.0 for QCS, and 105.98 on Walker2d-medium-replay, compared with 94.1 for QCS and 98.5 for QT. These results suggest that the proposed combination has practical value, with improvements extending beyond a single environment.

Critical weaknesses
  1. The core mechanism is not sufficiently supported. A fixed IQL critic does not simply evaluate the behavior policy, and joint training does not guarantee accurate out-of-distribution value estimates. The relationship between the multi-step target in Equation (3) and the current policy’s value is unclear. Section 6.5 provides incomplete ablation results and no direct value-error measurements. Moreover, exceeding the maximum dataset return does not establish trajectory stitching. These limitations leave the distinct contribution and mechanism beyond combining QCS and QT insufficiently demonstrated.

  2. Baseline comparisons and reporting inconsistencies weaken the conclusions. AntMaze results mix QT v0 with AQDT v2, while reproduced Adroit QT scores are substantially below the original results without sufficient explanation. Table 1 reports a CGDT average despite missing three scores; Appendix C.1’s “within 3.2 points” statement conflicts with Figure 6’s +6.4; and Section 6.5 confuses standard deviation with variance. These issues require correction and clearer experimental protocols.

Questions for authors
  1. Please clarify the exact evaluation and aggregation procedure for Table 2. With three seeds and only ten evaluation episodes per seed, assuming binary success scoring, the mean score should change in increments of approximately 3.333 points. How were 84.4, 87.8, and 85.6 obtained? Were results additionally averaged across multiple checkpoints or repeated evaluations?

  2. Which exact environment and dataset versions were used for the Adroit experiments? Why are the reproduced QT results substantially lower than those in the original paper? Please specify the training budgets, RTG settings, and model selection procedures for each method, and provide comparisons under the same protocol.

  3. Please provide the complete results for all four configurations across the three tasks in Section 6.5, and clarify whether they share the same IQL initialization, backbone, training budget, and hyperparameter search space. What direct evidence supports the claim that joint training improves alignment between the critic and the policy?

  4. How does Equation (3) handle termination, time-limit truncation, and the discrepancy between intermediate dataset actions and the target policy? Does Equation (5) use raw or preprocessed returns? When the maximum trajectory return is zero or negative, how does the method avoid division by zero or a reversal of the Q-guidance direction?

Reviewer_GBPeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Adaptive Q-Guided Decision Transformer (AQDT) for offline reinforcement learning, aiming to improve trajectory stitching from suboptimal demonstrations. The method combines trajectory-aware Q-guidance with a co-trained critic to balance imitation learning and value-based policy improvement. Experiments on D4RL benchmarks demonstrate competitive performance, particularly on datasets dominated by suboptimal trajectories.

Critical strengths

1.The paper addresses an important challenge in offline RL by combining the stability of sequence modeling with the policy improvement capabilities of value-based methods. 2.The proposed method is straightforward to implement and achieves promising performance improvements across multiple D4RL benchmarks.

Critical weaknesses

1.The proposed method essentially combines trajectory-aware Q-guidance from QCS with critic co-training from QT. These two components have already been explored in previous works, and their combination appears relatively straightforward, without introducing substantial theoretical insights or fundamentally new mechanisms. The authors should better clarify the technical challenges and novel contributions beyond combining existing approaches. 2. The authors argue that co-training the critic with the evolving policy improves value estimation and policy learning. However, in offline RL, simply updating the critic alongside the policy cannot prevent out-of-distribution (OOD) extrapolation errors. As the policy deviates from the behavior distribution, inaccurate Q-values may be further amplified through bootstrapping, potentially leading to unstable training. The paper lacks sufficient theoretical analysis or experimental evidence to demonstrate how the proposed mechanism effectively addresses this fundamental issue. 3. The proposed method determines Q-guidance strength based on the overall trajectory return. However, low-return trajectories may still contain locally optimal actions, while high-return trajectories may include suboptimal decisions. Therefore, uniformly weighting all actions within a trajectory may suppress useful behaviors or introduce unnecessary corrections. More fine-grained alternatives, such as state-level or advantage-based weighting, should be investigated. 4. The authors mainly use improvements in cumulative returns, including outperforming the best trajectories in the dataset, as evidence of successful trajectory stitching. However, higher returns do not necessarily imply that the policy successfully combines useful segments from different trajectories. More direct trajectory visualizations and quantitative stitching analyses are needed to support this claim. Furthermore, some AntMaze baselines, particularly QT, are evaluated using different benchmark versions, raising concerns about the fairness of the comparisons.

Questions for authors

Please refer to the Weaknesses section for my questions and concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.