- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work examines the role of the value-learning objective when using transformer-based critics in offline reinforcement learning (RL). A prior observation in [1] states that the reason for performance degradation with scaled transformer-based critics is attention collapse, where attention distributions over input tokens become highly concentrated, as reflected in low attention entropy. In this work, the authors argue that such an observation is a function of the value learning objective used. For instance, changing the value learning objective from standard Q-learning to in-sample learning removes such correlation between performance drop and attention entropy collapse. Since the in-sample learning objective requires an explicitly learned value function, the authors have studied various architectural configurations for both the state-value (V-function) and state-action value function (Q-function). The outcome of the study is that using a transformer-based architecture for the value function while using an MLP architecture for the Q-function would result in the best performance for their proposed method. Consequently, the proposed method is called state-Value Transformer Implicit Q-Learning or VTIQL for short. Evaluating this new method on environments from OGBench [2] shows that the proposed method outperforms other baselines on 7 out of the 9 environments, including the TQL [1] algorithm. Moreover, relying on a transformer architecture for the value function while using an MLP for the Q-function allows for faster training updates and lower peak memory usage.
[1] Perry Dong, Kuo-Han Hung, Alexander Swerdlow, Dorsa Sadigh, and Chelsea Finn. TQL: Scaling q-functions with transformers by preventing attention collapse. In the Forty-third International Conference on Machine Learning, 2026.
[2] Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. In International Conference on Learning Representations, 2025.
Critical strengths
- The paper is written clearly and is easy to follow.
- The main experimental results are dense, with a good number of datasets and environments, as well as baselines.
- The experimental results indeed show the potential of the proposed method in comparison to other baselines.
- I find the effect of using a transformer-based architecture for the value function on memory and update speed a very interesting side effect, and it aligns well with the performance gain, too.
Critical weaknesses
-
One crucial concern about this work is the limited number of environments used to understand the need to change the value-learning objective. The authors have used only one environment (cube-double) for the studies in Figures 1 and 2. I appreciate the consistency in using the same environment everywhere, yet drawing an observation statement or any insights from a single environment is very limiting and could be misleading. Given that the main results are conducted on 9 environments, I would have expected at least 3 diverse environments for these studies, since I understand such studies could be expensive in terms of compute. This is a very crucial point for me, as the whole motivation could be relying on flawed studies. Furthermore, it is not stated how many seeds were used for these experiments, especially since the results in Figure 2 (a) have no confidence intervals for the bar plots.
-
I also have concerns regarding the insights drawn from this work. I believe the work does not give enough insight into why in-sample learning is a better objective for transformer-based architectures. The connections between performance, learning objective, and attention memory collapse are blurry. The general issue of OOD actions for offline RL is already well-known, and IQL-style is known to be effective. The question that I believe needs further answering is the clear relation between the stability of the learning objective, like in-sample learning, and the transformer architecture. My point is that this part needs more explanation to share more fruitful insights aside from the stated empirical evidence.
-
The limitations are not stated. I believe the authors should share their view of their method's limitations.
Questions for authors
- Following Weakness point 1, I would like to ask the reviewer for the reason for conducting the studies in Figures 1 and 2 in only one environment. If it is due to computational constraints, I believe it is still crucial to show the same pattern on a smaller subset, at least 9 environments discussed in the main results.
- How many seeds were used in the experiments of Figures 1 and 2?
- In Figure 2, what are the sizes of the models used? I believe consistency between model sizes for the Transformer and MLP would eliminate the contribution of model capacity, which might lead to misleading results.
- Could you provide more insight into why the in-sample learning objective is improving performance over the standard Q-learning objective?
- In Figure 3, scaling the model size does not seem to be reflected in the performance; why is that? In addition, which model exactly was scaled: only the transformer or also the MLP (Q-function)?
- Could you please state the limitations of the proposed method from your point of view?
9 comments
Bellman Backup Variants
Our central insight is that the value learning objective is a key factor in the effectiveness of Transformer-based critics. In-sample learning avoids directly incorporating potentially overestimated Q-values for actions generated by the learned actor into the bootstrap targets. With the Transformer-based Q-function architecture and FQL policy extraction procedure held fixed at each model size, in-sample learning achieves higher mean success rates across the evaluated environments in Figure 1. These improvements are obtained without explicit attention entropy control.
Prior work [1] associates declining performance when scaling Transformer-based critics under the standard Q-learning backup with attention collapse. Although FQL-based policy extraction [2] regularizes the actor through distillation from a behavior cloning flow policy, the standard Q-learning backup still constructs targets using Q-values of actions generated by the learned actor. Consequently, the supervision used to train the Transformer-based critic depends on both evolving critic estimates and changes in actions generated by the learned policy. This motivates us to investigate whether the relationship between attention entropy and policy performance depends on Bellman target construction.
In-sample learning instead constructs the Q-function target as , with the V-function learned through expectile regression on Q-values of dataset actions. This removes the direct dependence of critic targets on actions generated by the learned actor, although these targets still depend on evolving critic estimates. For a Transformer-based Q-function, changes in the bootstrap targets affect the TD residual and therefore the gradients used to update the attention parameters. Actor updates alone do not cause such changes under the in-sample objective, removing one source of variation in this learning signal. In-sample learning removes a direct pathway through which potential overestimation of Q-values for actions generated by the learned actor enters this learning signal, providing a plausible explanation for the observed performance improvements.
To investigate this distinction, Figures 1, 12, and 13 in the revised paper compare the Q-learning and in-sample backups across Transformer-based critic sizes. At each size, both formulations use the same Transformer-based Q-function architecture and FQL policy extraction procedure. The in-sample formulation additionally uses an MLP-based V-function. Across cube-double, humanoidmaze-medium, and puzzle-4x4, in-sample learning achieves higher mean success rates than Q-learning at all tested model sizes. On cube-double, increasing the Transformer-based critic size under the Q-learning backup is accompanied by lower success rates and lower attention entropy toward the end of training. Under the in-sample backup, attention entropy also decreases with critic size, but success rates do not decrease monotonically. The additional experiments show that lower attention entropy need not imply lower performance: on humanoidmaze-medium, increasing the in-sample critic size from 1.6M to 6.6M decreases attention entropy from 1.95 to 1.71 while increasing the mean success rate from 25.93% to 34.20%. Therefore, these results support the effectiveness of in-sample Bellman backups for Transformer-based critics across the evaluated environments, even without attention entropy control.
Scaling the Model Size
The key observation in Figure 3 is that VTIQL achieves high performance even at the smallest model size and maintains this performance as model capacity increases, highlighting its parameter efficiency. "Model size" refers specifically to the learned transformer-based critic: the learned Q-function for FQL, TQL, and QTIQL, and the transformer-based V-function for VTIQL. Target Q-networks are excluded from the reported parameter counts. For VTIQL, only the Transformer-based V-function is scaled; the MLP-based Q-function is kept fixed.
In Figure 3, FQL and QTIQL show no consistent improvement as critic capacity increases, and both remain below VTIQL across all evaluated model sizes. TQL, even with a heuristically specified attention entropy target, improves at some larger model sizes but consistently underperforms VTIQL. At the smallest model size, VTIQL achieves approximately 2.5 times TQL’s average success rate across the three environments. Moreover, VTIQL with its smallest critic achieves a higher average success rate than any of the three baselines at any tested model size. Increasing baseline model capacity does not close the performance gap to VTIQL. VTIQL therefore demonstrates strong parameter efficiency, as even its smallest critic achieves a higher average success rate than any baseline at any tested model size.
Discussion and Future Work
Our evaluation focuses on nine state-based environments for reward-based, single-task offline RL in OGBench. Extending the proposed Transformer-based critic architecture to goal-conditioned reinforcement learning and settings with sparser reward feedback requires further investigation. We therefore plan to develop a more general architecture that accommodates different task specifications and reward structures while retaining the benefits of in-sample value learning.
Clarification of Experimental Details
Random Seeds: All experiments in Figures 1 and 2, as well as the additional experiments in Figures 9–13, were conducted using three random seeds.
Confidence Intervals: We have updated Figure 2(a) to include confidence intervals to better illustrate the variability across runs.
References
[1] Dong, P., et al. "TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapse." ICML, 2026.
[2] Park, S., et al. "Flow Q-Learning." ICML, 2025.
We appreciate your time and feedback on our paper. We address your questions below and appreciate the chance to strengthen our results.
Generalizability of Objective Findings
We extend the Bellman backup ablation to puzzle-4x4 (Figure 12) and humanoidmaze-medium (Figure 13) in the revised paper. For these additional experiments, we train each Bellman backup variant with three random seeds for each model size (0.4M, 1.6M, and 6.6M parameters) and each of the five tasks in each environment.
Consistent with Figure 1, in-sample learning achieves higher mean success rates than Q-learning at every tested model size in both additional environments. The in-sample results also support the observation that lower attention entropy is not necessarily accompanied by lower performance. On humanoidmaze-medium, increasing the model size from 1.6M to 6.6M reduces attention entropy from 1.95 to 1.71 while increasing the success rate from 25.93% to 34.20%. On puzzle-4x4, in-sample learning maintains higher mean success rates than Q-learning across all tested sizes as its attention entropy decreases from 2.21 to 1.21. These results show that in-sample Bellman backups remain effective across maze navigation and robotic manipulation tasks, even as attention becomes more concentrated.
Generalizability of Architectural Findings
We compare all four MLP/Transformer combinations on puzzle-4x4 (Figure 9) and humanoidmaze-medium (Figure 10) in the revised paper. For each task, we compute the held-out TD MSE for all four combinations using the same set of 8,192 transitions from a held-out validation split. Consistent with the success rate comparison in Figure 2, the combination of a Transformer-based V-function and an MLP-based Q-function achieves the highest mean success rate in both environments, 40.00% and 75.39%, respectively.
Replacing the MLP-based V-function with a Transformer reduces the mean held-out TD MSE under both Q-function architectures in both environments, indicating closer agreement with each model’s own bootstrap targets. However, replacing the MLP-based Q-function with a Transformer lowers the mean success rate under both V-function architectures, even in settings where it reduces TD MSE. These observations support our architectural design across the evaluated maze navigation and robotic manipulation tasks.
To control for differences in parameter counts, we additionally compare all four MLP/Transformer combinations on cube-double (Figure 11) with both MLP-based and Transformer-based critics containing approximately 6.6M parameters each. Thus, all four configurations have approximately 19.8M total critic parameters, including the Q-function, V-function, and target Q-network.
Under these matched parameter counts, the combination of a Transformer-based V-function and an MLP-based Q-function achieves the highest mean success rate of 60.07%, compared with 16.07%, 47.33%, and 42.40% for the MLP/MLP, Transformer/MLP, and Transformer/Transformer configurations, respectively, where the configurations are listed in Q/V order. This combination also achieves the lowest mean held-out TD MSE. Replacing the MLP-based V-function with a Transformer reduces the mean held-out TD MSE by 71.6% under the MLP-based Q-function and by 56.1% under the Transformer-based Q-function, indicating closer agreement with each model’s own bootstrap targets. These results further support our architectural design under matched parameter counts and show that its advantage is not solely attributable to differences in the number of critic parameters.