acceptodds
Under review as a conference paper at ICLR 2027

State-Value Transformers with In-Sample Bellman Backups

Abstract

Recent advances in Transformer-based critics have shown the benefits of scaling value functions in reinforcement learning (RL). However, these methods rely on specialized architectures or attention entropy control to scale effectively. In this paper, we identify the value learning objective as a key factor in the effectiveness of Transformer-based critics and demonstrate that in-sample learning improves performance without attention entropy control. Building on these findings, we propose State-Value Transformer Implicit Q-Learning (VTIQL), a method that combines a Transformer-based *V*-function with an MLP-based *Q*-function under in-sample learning. Extensive numerical experiments on OGBench demonstrate that VTIQL consistently outperforms prior offline RL methods, achieving the best performance on **7** out of 9 environments. Notably, VTIQL exhibits high performance on robotic manipulation tasks and maze navigation tasks with high-dimensional states, whereas prior methods struggle to achieve high performance in both task categories. VTIQL also performs strongly with a compact Transformer-based -function and remains robust across a wide range of model sizes.

Primary area:
reinforcement learning
Keywords:
State-Value Transformer, In-sample Bellman Backup, Offline Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_mMrnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work examines the role of the value-learning objective when using transformer-based critics in offline reinforcement learning (RL). A prior observation in [1] states that the reason for performance degradation with scaled transformer-based critics is attention collapse, where attention distributions over input tokens become highly concentrated, as reflected in low attention entropy. In this work, the authors argue that such an observation is a function of the value learning objective used. For instance, changing the value learning objective from standard Q-learning to in-sample learning removes such correlation between performance drop and attention entropy collapse. Since the in-sample learning objective requires an explicitly learned value function, the authors have studied various architectural configurations for both the state-value (V-function) and state-action value function (Q-function). The outcome of the study is that using a transformer-based architecture for the value function while using an MLP architecture for the Q-function would result in the best performance for their proposed method. Consequently, the proposed method is called state-Value Transformer Implicit Q-Learning or VTIQL for short. Evaluating this new method on environments from OGBench [2] shows that the proposed method outperforms other baselines on 7 out of the 9 environments, including the TQL [1] algorithm. Moreover, relying on a transformer architecture for the value function while using an MLP for the Q-function allows for faster training updates and lower peak memory usage.

[1] Perry Dong, Kuo-Han Hung, Alexander Swerdlow, Dorsa Sadigh, and Chelsea Finn. TQL: Scaling q-functions with transformers by preventing attention collapse. In the Forty-third International Conference on Machine Learning, 2026.

[2] Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. In International Conference on Learning Representations, 2025.

Critical strengths
  • The paper is written clearly and is easy to follow.
  • The main experimental results are dense, with a good number of datasets and environments, as well as baselines.
  • The experimental results indeed show the potential of the proposed method in comparison to other baselines.
  • I find the effect of using a transformer-based architecture for the value function on memory and update speed a very interesting side effect, and it aligns well with the performance gain, too.
Critical weaknesses
  1. One crucial concern about this work is the limited number of environments used to understand the need to change the value-learning objective. The authors have used only one environment (cube-double) for the studies in Figures 1 and 2. I appreciate the consistency in using the same environment everywhere, yet drawing an observation statement or any insights from a single environment is very limiting and could be misleading. Given that the main results are conducted on 9 environments, I would have expected at least 3 diverse environments for these studies, since I understand such studies could be expensive in terms of compute. This is a very crucial point for me, as the whole motivation could be relying on flawed studies. Furthermore, it is not stated how many seeds were used for these experiments, especially since the results in Figure 2 (a) have no confidence intervals for the bar plots.

  2. I also have concerns regarding the insights drawn from this work. I believe the work does not give enough insight into why in-sample learning is a better objective for transformer-based architectures. The connections between performance, learning objective, and attention memory collapse are blurry. The general issue of OOD actions for offline RL is already well-known, and IQL-style is known to be effective. The question that I believe needs further answering is the clear relation between the stability of the learning objective, like in-sample learning, and the transformer architecture. My point is that this part needs more explanation to share more fruitful insights aside from the stated empirical evidence.

  3. The limitations are not stated. I believe the authors should share their view of their method's limitations.

Questions for authors
  1. Following Weakness point 1, I would like to ask the reviewer for the reason for conducting the studies in Figures 1 and 2 in only one environment. If it is due to computational constraints, I believe it is still crucial to show the same pattern on a smaller subset, at least 9 environments discussed in the main results.
  2. How many seeds were used in the experiments of Figures 1 and 2?
  3. In Figure 2, what are the sizes of the models used? I believe consistency between model sizes for the Transformer and MLP would eliminate the contribution of model capacity, which might lead to misleading results.
  4. Could you provide more insight into why the in-sample learning objective is improving performance over the standard Q-learning objective?
  5. In Figure 3, scaling the model size does not seem to be reflected in the performance; why is that? In addition, which model exactly was scaled: only the transformer or also the MLP (Q-function)?
  6. Could you please state the limitations of the proposed method from your point of view?
9 comments

We appreciate your time and feedback on our paper. We address your questions below and appreciate the chance to strengthen our results.


Generalizability of Objective Findings

We extend the Bellman backup ablation to puzzle-4x4 (Figure 12) and humanoidmaze-medium (Figure 13) in the revised paper. For these additional experiments, we train each Bellman backup variant with three random seeds for each model size (0.4M, 1.6M, and 6.6M parameters) and each of the five tasks in each environment.

EnvironmentMetricBellman Backup0.4M1.6M6.6M
humanoidmaze-mediumSuccess (%)Q-learning3.333.202.80
In-sample44.6725.9334.20
Attention entropyQ-learning2.031.821.29
In-sample1.921.951.71
puzzle-4x4Success (%)Q-learning14.0016.1322.67
In-sample29.3335.3332.27
Attention entropyQ-learning2.341.611.01
In-sample2.211.941.21

Consistent with Figure 1, in-sample learning achieves higher mean success rates than Q-learning at every tested model size in both additional environments. The in-sample results also support the observation that lower attention entropy is not necessarily accompanied by lower performance. On humanoidmaze-medium, increasing the model size from 1.6M to 6.6M reduces attention entropy from 1.95 to 1.71 while increasing the success rate from 25.93% to 34.20%. On puzzle-4x4, in-sample learning maintains higher mean success rates than Q-learning across all tested sizes as its attention entropy decreases from 2.21 to 1.21. These results show that in-sample Bellman backups remain effective across maze navigation and robotic manipulation tasks, even as attention becomes more concentrated.


Generalizability of Architectural Findings

We compare all four MLP/Transformer combinations on puzzle-4x4 (Figure 9) and humanoidmaze-medium (Figure 10) in the revised paper. For each task, we compute the held-out TD MSE for all four combinations using the same set of 8,192 transitions from a held-out validation split. Consistent with the success rate comparison in Figure 2, the combination of a Transformer-based V-function and an MLP-based Q-function achieves the highest mean success rate in both environments, 40.00% and 75.39%, respectively.

EnvironmentQ ArchitectureV ArchitectureSuccess Rate (%)Held-out TD MSE
humanoidmaze-mediumMLPMLP51.07 ± 4.509.56 ± 3.13
MLPTransformer75.39 ± 2.108.18 ± 2.42
TransformerMLP41.93 ± 12.337.84 ± 2.68
TransformerTransformer28.98 ± 19.395.72 ± 2.62
puzzle-4x4MLPMLP36.27 ± 4.712353.91 ± 88.28
MLPTransformer40.00 ± 4.472333.80 ± 283.56
TransformerMLP33.80 ± 4.332673.11 ± 283.65
TransformerTransformer16.93 ± 4.301877.09 ± 3904.33

Replacing the MLP-based V-function with a Transformer reduces the mean held-out TD MSE under both Q-function architectures in both environments, indicating closer agreement with each model’s own bootstrap targets. However, replacing the MLP-based Q-function with a Transformer lowers the mean success rate under both V-function architectures, even in settings where it reduces TD MSE. These observations support our architectural design across the evaluated maze navigation and robotic manipulation tasks.

To control for differences in parameter counts, we additionally compare all four MLP/Transformer combinations on cube-double (Figure 11) with both MLP-based and Transformer-based critics containing approximately 6.6M parameters each. Thus, all four configurations have approximately 19.8M total critic parameters, including the Q-function, V-function, and target Q-network.

EnvironmentQ ArchitectureV ArchitectureSuccess Rate (%)Held-out TD MSE
cube-doubleMLPMLP16.07 ± 3.53249.75 ± 28.82
MLPTransformer60.07 ± 1.1370.87 ± 10.49
TransformerMLP47.33 ± 5.20220.92 ± 32.03
TransformerTransformer42.40 ± 7.4097.03 ± 70.57

Under these matched parameter counts, the combination of a Transformer-based V-function and an MLP-based Q-function achieves the highest mean success rate of 60.07%, compared with 16.07%, 47.33%, and 42.40% for the MLP/MLP, Transformer/MLP, and Transformer/Transformer configurations, respectively, where the configurations are listed in Q/V order. This combination also achieves the lowest mean held-out TD MSE. Replacing the MLP-based V-function with a Transformer reduces the mean held-out TD MSE by 71.6% under the MLP-based Q-function and by 56.1% under the Transformer-based Q-function, indicating closer agreement with each model’s own bootstrap targets. These results further support our architectural design under matched parameter counts and show that its advantage is not solely attributable to differences in the number of critic parameters.


Bellman Backup Variants

Our central insight is that the value learning objective is a key factor in the effectiveness of Transformer-based critics. In-sample learning avoids directly incorporating potentially overestimated Q-values for actions generated by the learned actor into the bootstrap targets. With the Transformer-based Q-function architecture and FQL policy extraction procedure held fixed at each model size, in-sample learning achieves higher mean success rates across the evaluated environments in Figure 1. These improvements are obtained without explicit attention entropy control.

Prior work [1] associates declining performance when scaling Transformer-based critics under the standard Q-learning backup with attention collapse. Although FQL-based policy extraction [2] regularizes the actor through distillation from a behavior cloning flow policy, the standard Q-learning backup still constructs targets using Q-values of actions generated by the learned actor. Consequently, the supervision used to train the Transformer-based critic depends on both evolving critic estimates and changes in actions generated by the learned policy. This motivates us to investigate whether the relationship between attention entropy and policy performance depends on Bellman target construction.

In-sample learning instead constructs the Q-function target as r+γV(s′)r+\gamma V(s'), with the V-function learned through expectile regression on Q-values of dataset actions. This removes the direct dependence of critic targets on actions generated by the learned actor, although these targets still depend on evolving critic estimates. For a Transformer-based Q-function, changes in the bootstrap targets affect the TD residual and therefore the gradients used to update the attention parameters. Actor updates alone do not cause such changes under the in-sample objective, removing one source of variation in this learning signal. In-sample learning removes a direct pathway through which potential overestimation of Q-values for actions generated by the learned actor enters this learning signal, providing a plausible explanation for the observed performance improvements.

To investigate this distinction, Figures 1, 12, and 13 in the revised paper compare the Q-learning and in-sample backups across Transformer-based critic sizes. At each size, both formulations use the same Transformer-based Q-function architecture and FQL policy extraction procedure. The in-sample formulation additionally uses an MLP-based V-function. Across cube-double, humanoidmaze-medium, and puzzle-4x4, in-sample learning achieves higher mean success rates than Q-learning at all tested model sizes. On cube-double, increasing the Transformer-based critic size under the Q-learning backup is accompanied by lower success rates and lower attention entropy toward the end of training. Under the in-sample backup, attention entropy also decreases with critic size, but success rates do not decrease monotonically. The additional experiments show that lower attention entropy need not imply lower performance: on humanoidmaze-medium, increasing the in-sample critic size from 1.6M to 6.6M decreases attention entropy from 1.95 to 1.71 while increasing the mean success rate from 25.93% to 34.20%. Therefore, these results support the effectiveness of in-sample Bellman backups for Transformer-based critics across the evaluated environments, even without attention entropy control.


Scaling the Model Size

The key observation in Figure 3 is that VTIQL achieves high performance even at the smallest model size and maintains this performance as model capacity increases, highlighting its parameter efficiency. "Model size" refers specifically to the learned transformer-based critic: the learned Q-function for FQL, TQL, and QTIQL, and the transformer-based V-function for VTIQL. Target Q-networks are excluded from the reported parameter counts. For VTIQL, only the Transformer-based V-function is scaled; the MLP-based Q-function is kept fixed.

In Figure 3, FQL and QTIQL show no consistent improvement as critic capacity increases, and both remain below VTIQL across all evaluated model sizes. TQL, even with a heuristically specified attention entropy target, improves at some larger model sizes but consistently underperforms VTIQL. At the smallest model size, VTIQL achieves approximately 2.5 times TQL’s average success rate across the three environments. Moreover, VTIQL with its smallest critic achieves a higher average success rate than any of the three baselines at any tested model size. Increasing baseline model capacity does not close the performance gap to VTIQL. VTIQL therefore demonstrates strong parameter efficiency, as even its smallest critic achieves a higher average success rate than any baseline at any tested model size.


Discussion and Future Work

Our evaluation focuses on nine state-based environments for reward-based, single-task offline RL in OGBench. Extending the proposed Transformer-based critic architecture to goal-conditioned reinforcement learning and settings with sparser reward feedback requires further investigation. We therefore plan to develop a more general architecture that accommodates different task specifications and reward structures while retaining the benefits of in-sample value learning.


Clarification of Experimental Details

Random Seeds: All experiments in Figures 1 and 2, as well as the additional experiments in Figures 9–13, were conducted using three random seeds.

Confidence Intervals: We have updated Figure 2(a) to include confidence intervals to better illustrate the variability across runs.


References

[1] Dong, P., et al. "TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapse." ICML, 2026.

[2] Park, S., et al. "Flow Q-Learning." ICML, 2025.

Reviewer_XMzrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Auther first show that attention entropy reduction does not necessarily result performance degradation for in sample learning. They proposed VTIQL which use V function transformer with MLP based Q function in the IQL framework. The result shows the proposed structure result in performance improvement.

Critical strengths
  1. This paper do study a critical area with some noval network structure change
  2. The proposed architecture is simple but with performance improvement
Critical weaknesses
  1. Overall, auther did show by applying a transformer V function the performance improves but auther do not isolate whether the gain comes from state-only modeling, avoidance of OOD action queries, expectile regression, or differences in model capacity. Since several factors change simultaneously, the current results support the architectural choice but not the proposed mechanism.
  2. For Table 2, auther state " total critic parameter counts can differ across methods once target networks are included" so how can the computational efficiency still fair?
Questions for authors
  1. in line 130, based on auther defination of V. We can get Vπ=Ea∼π[Qπ(s,a)]V^{\pi} = E_{a ∼ \pi} [Q^{\pi}(s,a)]. However, the later IQL objective (line 205) the VψV_\psi is not the VπV^{\pi} since IQL objective samples a from the offline dataset and therefore learns a conditional expectile of Q(s,a). The authors should distinguish these objects notationally.
  2. Figure 1, auther use very similar color and the shaded area is really hard to distinguish.
  3. Why V function can help more than Q function? Can this be separated from the absence of action inputs and from the expectile-regression objective?
6 comments

Thank you for your positive evaluation of our paper and your insightful feedback. We address your questions below.


Transformer-based V-Function

Our design combines a Transformer-based V-function with an MLP-based Q-function while retaining the IQL value-learning formulation. In IQL [1], a state-only V-function is trained through expectile regression on Q-values of dataset actions and used to construct in-sample Bellman targets without querying out-of-dataset actions during critic bootstrapping.

To evaluate the effect of the V-function architecture, we compare MLP-based and Transformer-based V-functions in Section 4.2 while keeping the MLP-based Q-function architecture and FQL policy extraction procedure fixed. Both configurations use state-only V-function inputs, the same expectile regression objective, and the same in-sample Bellman backup. These comparisons therefore show the additional benefit of a Transformer-based V-function within the same IQL formulation. This improvement also persists in our additional experiment with matched parameter counts.

Additionally, we compare all four MLP/Transformer combinations on puzzle-4x4 (Figure 9) and humanoidmaze-medium (Figure 10) in the revised paper. For each task, we compute the held-out TD MSE for all four combinations using the same set of 8,192 transitions from a held-out validation split. Consistent with the success rate comparison in Figure 2, the combination of a Transformer-based V-function and an MLP-based Q-function achieves the highest mean success rate in both environments, 40.00% and 75.39%, respectively.

EnvironmentQ ArchitectureV ArchitectureSuccess Rate (%)Held-out TD MSE
humanoidmaze-mediumMLPMLP51.07 ± 4.509.56 ± 3.13
MLPTransformer75.39 ± 2.108.18 ± 2.42
TransformerMLP41.93 ± 12.337.84 ± 2.68
TransformerTransformer28.98 ± 19.395.72 ± 2.62
puzzle-4x4MLPMLP36.27 ± 4.712353.91 ± 88.28
MLPTransformer40.00 ± 4.472333.80 ± 283.56
TransformerMLP33.80 ± 4.332673.11 ± 283.65
TransformerTransformer16.93 ± 4.301877.09 ± 3904.33

Replacing the MLP-based V-function with a Transformer reduces the mean held-out TD MSE under both Q-function architectures in both environments, indicating closer agreement with each model’s own bootstrap targets. However, replacing the MLP-based Q-function with a Transformer lowers the mean success rate under both V-function architectures, even in settings where it reduces TD MSE. These observations support our architectural design across the evaluated maze navigation and robotic manipulation tasks.

To control for differences in parameter counts, we additionally compare all four MLP/Transformer combinations on cube-double (Figure 11) with both MLP-based and Transformer-based critics containing approximately 6.6M parameters each. Thus, all four configurations have approximately 19.8M total critic parameters, including the Q-function, V-function, and target Q-network.

EnvironmentQ ArchitectureV ArchitectureSuccess Rate (%)Held-out TD MSE
cube-doubleMLPMLP16.07 ± 3.53249.75 ± 28.82
MLPTransformer60.07 ± 1.1370.87 ± 10.49
TransformerMLP47.33 ± 5.20220.92 ± 32.03
TransformerTransformer42.40 ± 7.4097.03 ± 70.57

Under these matched parameter counts, the combination of a Transformer-based V-function and an MLP-based Q-function achieves the highest mean success rate of 60.07%, compared with 16.07%, 47.33%, and 42.40% for the MLP/MLP, Transformer/MLP, and Transformer/Transformer configurations, respectively, where the configurations are listed in Q/V order. This combination also achieves the lowest mean held-out TD MSE. Replacing the MLP-based V-function with a Transformer reduces the mean held-out TD MSE by 71.6% under the MLP-based Q-function and by 56.1% under the Transformer-based Q-function, indicating closer agreement with each model’s own bootstrap targets. These results further support our architectural design under matched parameter counts and show that its advantage is not solely attributable to differences in the number of critic parameters.


Fair Comparison of Computational Efficiency

We conduct additional efficiency comparisons with approximately matched total critic parameter counts, including parameters in target networks. We keep the Transformer critic size fixed within each comparison and increase the capacity of VTIQL’s MLP-based Q-function and its target network to match the overall parameter budgets more closely.

Total critic parameters (M)Transformer sizeMethodV (M)Q (M)Target Q (M)Update time (ms) ↓Relative update time ↓Peak Memory (GiB) ↓
≈13M6.6MVTIQL6.63.53.549.91.00$\times$1.40
QTIQL0.86.66.674.11.48$\times$2.18
TQL—6.66.668.41.37$\times$2.01
≈53M26MVTIQL2613.513.5135.71.00$\times$3.20
QTIQL0.82626169.81.25$\times$4.98
TQL—2626165.01.22$\times$4.69

At the 26M Transformer size, VTIQL has 53.0M total critic parameters, only 0.4% more than QTIQL (52.8M) and 1.9% more than TQL (52.0M). With comparable parameter counts, VTIQL reduces update time by 20.1% relative to QTIQL and 17.8% relative to TQL. It also reduces peak memory usage by 35.7% and 31.8%, respectively. The comparison at the 6.6M Transformer size shows the same pattern. The observed memory advantage is consistent with differences in policy gradient computation. VTIQL differentiates through an MLP-based Q-function, whereas QTIQL and TQL differentiate through Transformer-based Q-functions, resulting in different computational graphs and requirements for intermediate activations and temporary buffers. These results demonstrate that VTIQL’s computational advantage persists under approximately matched total parameter counts and is not solely attributable to having fewer critic parameters.

To further examine the contribution of target Q-network computation, we conduct additional experiments with approximately matched total critic parameter counts.

Total critic parameters (M)Transformer sizeMethodV (M)Q (M)Target Q (M)Update time (ms) ↓Relative update time ↓Peak Memory (GiB) ↓Q inference time (ms) ↓Relative Q time ↓
≈19.8M6.6MVTIQL6.66.66.659.11.00$\times$1.501.41.00$\times$
QTIQL6.66.66.686.61.47$\times$2.258.96.29$\times$
≈78.0–78.7M26MVTIQL25.826.126.1164.21.00$\times$3.645.41.00$\times$
QTIQL26.126.326.3206.81.26$\times$5.4924.64.54$\times$

VTIQL achieves update speedups of 1.26–1.47$\times$ over QTIQL while reducing peak memory usage by approximately 33%. Notably, MLP-based Q inference achieves speedups of 4.54–6.29$\times$ over Transformer-based Q inference. The measured Q inference time differences correspond to approximately 27–45% of the total update-time gaps, suggesting that avoiding a Transformer-based target Q-network contributes substantially to computational efficiency. The remaining gains may also reflect differences in trainable Q-network computation.


Clarity of Notations and Figures

Notation. We use VπV^{\pi} to denote the policy value defined in Section 3.1 and VψV_{\psi} to denote the parameterized state-value function trained using the IQL objective. Specifically, VψV_\psi is trained to estimate the conditional τ\tau-expectile of the target Q-values Qθ^(s,a)Q_{\hat\theta}(s,a) over dataset actions at each state. We have made this distinction explicit in lines 211–212 of the revised paper.

Figures. We have revised Figure 1 to use more distinct colors, making the curves and shaded regions easier to distinguish. We have also applied the same color scheme to the figures presenting the additional experiments in the revised paper.


References

[1] Kostrikov, I., et al. "Offline Reinforcement Learning with Implicit Q-Learning." ICLR, 2022.

Dear Reviewer XMzr,

Thank you for your thoughtful review. As the rebuttal period is ongoing, we would like to follow up on our response.

  • We have clarified that Section 4.2 demonstrates the additional benefit of a Transformer-based V-function within the same IQL formulation, with the MLP-based Q-function architecture and policy extraction procedure held fixed. Additionally, we have compared all four MLP/Transformer combinations on puzzle-4x4 (Figure 9) and humanoidmaze-medium (Figure 10). VTIQL achieves the highest mean success rate among the four configurations in both tasks.
  • To control for differences in parameter counts, we have compared all four MLP/Transformer combinations on cube-double (Figure 11), with both MLP-based and Transformer-based critics containing approximately 6.6M parameters each. Under these matched parameter counts, VTIQL achieves the highest mean success rate.
  • We have also conducted efficiency comparisons with approximately matched total critic parameter counts, including target networks. VTIQL maintains lower update time and peak memory usage than QTIQL and TQL. Additional comparisons against QTIQL show 1.26–1.47$\times$ update speedups and approximately 33% lower peak memory usage.
  • We have clarified the distinction between the policy value VπV^{\pi} and the parameterized state-value function VψV_{\psi} trained using the IQL objective, and revised the figure colors to improve readability.

Please let us know if the rebuttal has answered your questions. If the Reviewer does not have any additional concerns or questions, we would sincerely ask that the score be adjusted accordingly.

Yours sincerely,

The Authors

Reviewer_sYbDOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the relationship between Bellman backup objectives and the performance of Transformer-based critics in offline reinforcement learning. The authors observe that under in-sample learning, lower attention entropy does not necessarily correspond to worse performance, unlike the behavior observed under standard Q-learning. Motivated by this finding, the authors propose VTIQL, which combines a Transformer-based V-function with an MLP-based Q-function using IQL critic objectives and FQL policy extraction. The proposed method achieves strong performance on OGBench and demonstrates favorable computational efficiency.

Critical strengths
  1. Interesting empirical observations. The finding that decreasing attention entropy does not necessarily lead to worse performance under in-sample Bellman backups is interesting and provides a different perspective on the previously reported attention-collapse problem.

  2. Surprising architectural findings. The substantial performance advantage of combining a Transformer-based V-function with an MLP-based Q-function is surprising and potentially valuable, especially given that prior work has primarily focused on Transformer-based Q-functions.

  3. Comprehensive experiments and ablations. The paper evaluates the proposed method across diverse offline RL tasks, including maze navigation and robotic manipulation. The ablations on critic architectures, model sizes, and attention head configurations are informative and support the effectiveness of the proposed approach.

  4. Computational efficiency. The improvements in training speed and memory efficiency provide an additional practical advantage.

  5. Clear presentation. The paper is generally well written and organized, with clear experimental procedures and informative figures.

Critical weaknesses
  1. Limited intuition behind the architectural design. While Section 4.2 provides empirical evidence that using a Transformer-based V-function with an MLP-based Q-function achieves superior performance, the paper offers limited intuition for why this particular combination is preferable. The underlying reasons why Transformers benefit V-function learning but not Q-function learning remain unclear.

  2. Incomplete positioning relative to recent baselines. While Table 1 demonstrates strong performance against the selected baselines, Appendix B.4 includes several recent methods that substantially outperform VTIQL on certain environments. For instance, DriftQL achieves 60% on antmaze-giant compared with VTIQL's 15%, while GFP achieves 94% on antmaze-large compared with VTIQL's 71%. These results meaningfully qualify the paper's claims of superiority over prior offline RL methods. Although the authors acknowledge differences in methodological scope and evaluation protocols, the main benchmark table also incorporates previously reported results, making the separation of these comparisons less convincing.

  3. Computational efficiency comparisons are not fully controlled for model size. Table 2 demonstrates that VTIQL achieves faster training updates and lower memory usage than QTIQL and TQL at matched Transformer sizes. However, the total critic parameter counts differ substantially because VTIQL uses a smaller MLP-based target Q-network, while the other methods maintain Transformer-based target Q-networks. Consequently, some of the efficiency gains are expected from the architectural differences.

  4. Limited interpretation of the TD MSE results. Section 4.2 uses held-out TD MSE to evaluate Bellman consistency. However, the TD targets are constructed using each model's own learned V-function. Therefore, lower TD MSE does not necessarily indicate more accurate value estimates, but rather closer agreement with the model's own bootstrap targets.

Questions for authors
  1. Architectural motivation: Why does using a Transformer substantially improve V-function learning but provide less benefit, or even degrade performance, when used for the Q-function? Could the authors provide additional intuition or analysis regarding the different optimization characteristics, representational requirements, or extrapolation behavior of these functions?

  2. Generalizability of architectural findings: The four-way architectural comparison in Figure 2 is conducted only on cube-double. Could the authors extend this ablation to additional environments, particularly maze navigation tasks, to establish whether the architectural advantage generalizes across domains?

  3. Comparison with recent baselines: What motivates placing the comparisons with GFP, Q-Flow, DriftQL, and Value Flows exclusively in the appendix? Given that these methods outperform VTIQL on several tasks, it would be helpful to discuss these results more prominently in the main text and clarify their implications for the paper's empirical claims.

  4. Computational efficiency: Could the authors provide comparisons under matched total parameter counts or computational budgets, or further discuss how much of the observed efficiency improvement can be attributed to avoiding a Transformer-based target Q-network?

  5. Bellman consistency: Can the authors provide additional evidence connecting lower held-out TD MSE to more accurate value estimates or improved policy performance?

7 comments

Thank you for your positive evaluation of our paper and your insightful feedback. We address your questions below.


Generalizability of Architectural Findings

We compare all four MLP/Transformer combinations on puzzle-4x4 (Figure 9) and humanoidmaze-medium (Figure 10) in the revised paper. For each task, we compute the held-out TD MSE for all four combinations using the same set of 8,192 transitions from a held-out validation split. Consistent with the success rate comparison in Figure 2, the combination of a Transformer-based V-function and an MLP-based Q-function achieves the highest mean success rate in both environments, 40.00% and 75.39%, respectively.

EnvironmentQ ArchitectureV ArchitectureSuccess Rate (%)Held-out TD MSE
humanoidmaze-mediumMLPMLP51.07 ± 4.509.56 ± 3.13
MLPTransformer75.39 ± 2.108.18 ± 2.42
TransformerMLP41.93 ± 12.337.84 ± 2.68
TransformerTransformer28.98 ± 19.395.72 ± 2.62
puzzle-4x4MLPMLP36.27 ± 4.712353.91 ± 88.28
MLPTransformer40.00 ± 4.472333.80 ± 283.56
TransformerMLP33.80 ± 4.332673.11 ± 283.65
TransformerTransformer16.93 ± 4.301877.09 ± 3904.33

Replacing the MLP-based V-function with a Transformer reduces the mean held-out TD MSE under both Q-function architectures in both environments, indicating closer agreement with each model’s own bootstrap targets. However, replacing the MLP-based Q-function with a Transformer lowers the mean success rate under both V-function architectures, even in settings where it reduces TD MSE. These observations support our architectural design across the evaluated maze navigation and robotic manipulation tasks.


Comparison with Recent Baselines

Our research question is what value learning objective is suitable for Transformer-based critics and how the architectures of the Q- and V-functions affect performance. Accordingly, Table 1 includes baselines relevant to this question: in-sample learning methods (IQL and IFQL), FQL, whose policy extraction procedure we adopt, and methods with Transformer-based critics (PAC and TQL [1]). These baselines were selected to provide a fair and relevant comparison aligned with our research question.

In contrast, the baselines discussed in Appendix B.4 represent subsequent developments beyond FQL [2], introducing additional mechanisms or alternative formulations for value learning and policy optimization. GFP [3] introduces value-aware behavior cloning through Q-weighted flow matching and considers a conservative variant of the Bellman target. Q-Flow [4] introduces intermediate value learning and uses intermediate value gradients to guide policy optimization. DriftQL [5] replaces flow policy distillation with a drift loss combining attraction and repulsion. Value Flows [6] models return distributions through distributional flow matching, with bootstrapped conditional flow matching regularization and confidence weights based on return variance.

Despite these additional mechanisms, Table 6 demonstrates VTIQL’s competitiveness across OGBench environments. VTIQL achieves higher reported mean success rates than GFP in 6/8, Q-Flow in 5/8, DriftQL in 5/8, and Value Flows in 4/4 environments. Its strengths are particularly evident in robotic manipulation, where it achieves higher reported means than GFP, Q-Flow, and Value Flows in 4/4 environments, and than DriftQL in 3/4 environments. These results highlight the effectiveness of combining in-sample value learning with a Transformer-based V-function and an MLP-based Q-function, even without these additional mechanisms. This design may also accommodate complementary techniques from these recent methods, offering opportunities to broaden VTIQL’s applicability.

We placed these comparisons in Appendix B.4 to keep the main presentation focused on the value learning objectives and Q- and V-function architectures investigated in this work. Following the reviewer’s suggestion, we will consider how best to present these comparisons alongside the experimental results to clarify their implications for our empirical claims.


Computational Efficiency

We conduct additional efficiency comparisons with approximately matched total critic parameter counts, including parameters in target networks. We keep the Transformer critic size fixed within each comparison and increase the capacity of VTIQL’s MLP-based Q-function and its target network to match the overall parameter budgets more closely.

Total critic parameters (M)Transformer sizeMethodV (M)Q (M)Target Q (M)Update time (ms) ↓Relative update time ↓Peak Memory (GiB) ↓
≈13M6.6MVTIQL6.63.53.549.91.00$\times$1.40
QTIQL0.86.66.674.11.48$\times$2.18
TQL—6.66.668.41.37$\times$2.01
≈53M26MVTIQL2613.513.5135.71.00$\times$3.20
QTIQL0.82626169.81.25$\times$4.98
TQL—2626165.01.22$\times$4.69

At the 26M Transformer size, VTIQL has 53.0M total critic parameters, only 0.4% more than QTIQL (52.8M) and 1.9% more than TQL (52.0M). With comparable parameter counts, VTIQL reduces update time by 20.1% relative to QTIQL and 17.8% relative to TQL. It also reduces peak memory usage by 35.7% and 31.8%, respectively. The comparison at the 6.6M Transformer size shows the same pattern. The observed memory advantage is consistent with differences in policy gradient computation. VTIQL differentiates through an MLP-based Q-function, whereas QTIQL and TQL differentiate through Transformer-based Q-functions, resulting in different computational graphs and requirements for intermediate activations and temporary buffers. These results demonstrate that VTIQL’s computational advantage persists under approximately matched total parameter counts and is not solely attributable to having fewer critic parameters.

To further examine the contribution of target Q-network computation, we conduct additional experiments with approximately matched total critic parameter counts.

Total critic parameters (M)Transformer sizeMethodV (M)Q (M)Target Q (M)Update time (ms) ↓Relative update time ↓Peak Memory (GiB) ↓Q inference time (ms) ↓Relative Q time ↓
≈19.8M6.6MVTIQL6.66.66.659.11.00$\times$1.501.41.00$\times$
QTIQL6.66.66.686.61.47$\times$2.258.96.29$\times$
≈78.0–78.7M26MVTIQL25.826.126.1164.21.00$\times$3.645.41.00$\times$
QTIQL26.126.326.3206.81.26$\times$5.4924.64.54$\times$

VTIQL achieves update speedups of 1.26–1.47$\times$ over QTIQL while reducing peak memory usage by approximately 33%. Notably, MLP-based Q inference achieves speedups of 4.54–6.29$\times$ over Transformer-based Q inference. The measured Q inference time differences correspond to approximately 27–45% of the total update-time gaps, suggesting that avoiding a Transformer-based target Q-network contributes substantially to computational efficiency. The remaining gains may also reflect differences in trainable Q-network computation.


Architectural Motivation and Bellman Consistency

Our design is motivated by the different roles of the V- and Q-functions in in-sample learning. The V-function is trained through expectile regression on the target Q-values of dataset actions and is used to construct in-sample Bellman targets. The Q-function additionally supports policy improvement through action-dependent values and gradients. Following the FQL policy extraction procedure [2], we train a one-step policy to maximize Q-values while regularizing it through distillation from a behavioral cloning (BC) flow policy. Although critic learning uses dataset actions, policy extraction evaluates the Q-function at actions generated by the one-step policy, which may extend beyond the dataset support.

To examine both Bellman consistency and the effectiveness of the learned Q-functions for policy improvement, we extend the cube-double analysis in Section 4.2 with additional experiments on all five HumanoidMaze-medium tasks, using checkpoints saved after 1M training steps for each of three training seeds. We freeze the learned critics and repeat policy extraction using each method’s BC flow policy as a shared, frozen reference. For each task, training seed, and BC flow policy, we initialize a common one-step policy through 2,000 distillation-only updates. Starting from these identical policy parameters, we then perform 10,000 policy updates, matching minibatches, latent noise, and optimizer settings across the two critics. Each resulting policy is evaluated over 100 episodes using matched evaluation seeds. We vary the distillation weight over α∈10,30,60\alpha\in\\{10,30,60\\}, with α=30\alpha=30 specified as the primary comparison.

For Bellman consistency, we evaluate both methods on the same 200,000 held-out transitions per task. Following Section 4.2, we compute the mean squared TD residual over dataset transitions and both Q heads, using bootstrap targets constructed from each model’s own V-function. As shown in Figure 14 in the revised paper, VTIQL achieves lower held-out TD MSE in all 15 task–seed pairs, reducing the overall mean from 8.54 to 6.65. Under controlled policy extraction at α=30\alpha=30, mean success averaged over both BC flow policies increases from 43.5% to 66.1%. Thus, the configuration with lower held-out TD MSE also supports higher mean policy performance under matched extraction conditions.

Figure 15 in the revised paper presents these policy results separately for each BC flow policy and extends the comparison across the three distillation weights. VTIQL achieves higher mean success rates under both flow policies at every tested weight. At α=30\alpha=30, the mean success rates increases from 38.7% to 67.5% with the QTIQL BC flow policy and from 48.3% to 64.7% with the VTIQL BC flow policy. The performance advantage therefore cannot be attributed solely to the choice of BC flow policy used for distillation.

Consistent with Section 4.2, we interpret lower held-out TD MSE as closer agreement with the respective bootstrap targets, rather than a direct measure of true-value accuracy. The reversal observed in task 2 of Figure 14 also shows that lower TD error does not guarantee higher policy performance on every task. Nevertheless, these experiments provide additional evidence that VTIQL achieves both better held-out Bellman consistency and higher average policy performance. Together with the comparison of all four MLP/Transformer combinations in Section 4.2, these results support our design, which uses a Transformer-based V-function to construct in-sample Bellman targets and an MLP-based Q-function for policy improvement.


References

[1] Dong, P., et al. "TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapse." ICML, 2026.

[2] Park, S., et al. "Flow Q-Learning." ICML, 2025.

[3] Tiofack, F. N., et al. "Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learning." ICLR, 2026.

[4] Doo, J., et al. "Q-Flow: Stable and Expressive Reinforcement Learning with Flow-based Policy." ICML, 2026.

[5] Houssaini, A., et al. "Drift Q-Learning." NeurIPS, 2026.

[6] Dong, P., et al. "Value Flows." ICLR, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.