acceptodds
Under review as a conference paper at ICLR 2027

The Low-Rank Hankel Hypothesis in Reinforcement Learning

Abstract

Identifying and exploiting structure in reinforcement learning (RL) can guide exploration and improve sample efficiency, particularly when environment interactions are costly. We investigate the *low-rank Hankel hypothesis*: value-function sequences evaluated along trajectories of high-performing policies exhibit low effective Hankel rank, and can therefore be approximated by low-order linear recurrences. We provide empirical evidence for this structure in learned critic outputs across Classical Control, MuJoCo, and Atari. Theoretically, we show that such structure can arise under policies inducing geometrically mixing state chains. Motivated by these findings, we introduce Q-Hankeling, a simple method that regularizes Q-function learning using short trajectory chunks sampled from replay. The method learns linear recurrence coefficients jointly with the critic and penalizes recurrence residuals within these chunks. This provides a temporal-predictability inductive bias for value estimation, without explicitly constructing Hankel matrices or computing singular-value decompositions. We demonstrate how Q-Hankeling can be integrated into value-based methods, such as DQN and Efficient Rainbow, and actor–critic methods, such as TD3 and SAC. On Atari-100k, Q-Hankeling increases the mean human-normalized score from 0.463 to 0.530 across 26 games, an improvement of approximately 14%. Across the reported MuJoCo tasks, Q-Hankeling improves final return at 1M environment steps in seven of eight evaluated SAC/TD3 task-algorithm combinations.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Sample efficiency, Low-rank Hankel Matrices, Actor-Critic methods, Value-based methods, Regularization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3UNGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Q-Hankeling, which regularizes the Q-function approximator using an autoregressive proxy to encourage effective low Hankel-rank along the temporal axis. This is motivated by the observation that the Hankel rank (of the Q-values along a trajectory) tends to decrease over the course of training. The authors integrate Q-Hankeling into DQN, SAC, and TD3, and report improved performance across various environments in standard benchmarks.

Critical strengths

The paper identifies an interesting pattern that Q-values along trajectories tend to have decreasing effective Hankel rank during training across several environments. It also proposes a practical way to exploit this observation through a learned autoregressive penalty, avoiding explicit Hankel-matrix construction. The regularizer integrates readily into several established RL algorithms, and the reported gains across many evaluated settings support its practical utility.

Critical weaknesses

Since the Hankel matrices are constructed from learned critic outputs along trajectories, the observed rank decrease is entangled by properties of the function approximator, changes in the visited trajectories, or structure in the underlying policy values. The experiments do not isolate these explanations. An experiment using MC rollouts can partially address this. This also leaves the applicability of the proposed regularizer somewhat unclear. If the true Q-value sequences do not admit a low rank structure, is this regularization still beneficial?

More broadly, the RL interpretation of the regularizer remains underdeveloped. Although it explicitly encourages critic outputs to follow a shared linear recurrence, it is unclear what desirable property of value estimation or policy learning this penalty encourages. The reported performance gains demonstrate usefulness in several settings, but do not establish the underlying mechanism.

Questions for authors
  1. From an RL perspective (given the true q-values), what are the implications of the Hankel rank?
  2. What explains the observed low effective Hankel rank? Since the measurements use learned Q-functions under evolving policies, can the authors distinguish structure in true policy values from effects of function approximation and changes in visited trajectories?
3 comments

We thank the reviewer for their comments, which will help us improve the manuscript.

W1: Where does low-rank Hankel structure come from?

Explaining why low-rank Hankel structure emerges during critic learning remains open. As discussed in our response to Reviewer QoJW, W1, Theorem 4.1 motivates the recurrence penalty, while Theorem 4.2 investigates mixing as a possible explanation for the structure under a stationary policy and exact values. These insights provide useful guidance, but our rollout measurements alone do not disentangle true-value structure from function approximation and visitation effects. We refer the reviewer to that response for the supporting evidence and theoretical limitations, and to Q2 below for a Monte Carlo control on Classical Control. Whether the regularizer remains beneficial when true value sequences lack this structure is also unresolved.

W2: On the role of the regularizer

Q-Hankeling aims to encourage low-rank Hankel structure in critic predictions through a low-order recurrence penalty, motivated by Theorem 4.1. The Hankel diagnostics in Appendix E.4.3 provide empirical evidence that it encourages this structure on replay windows by the end of training, alongside the aggregate performance improvements reported in Tables 1 and 2 of the manuscript. These findings support the usefulness of encouraging low-rank Hankel structure through the recurrence penalty motivated by Theorem 4.1. In Q1, we discuss how this structural bias may improve sample efficiency by reducing the effective complexity of critic learning. We will clarify this interpretation in the revised manuscript.

Q1: Implications of low-rank Hankel structure

For a fixed policy, low Hankel rank means that its values along a trajectory admit a compact recurrence description. Under the assumptions of Theorem 4.1, approximate low rank is equivalent to an approximate low-order linear recurrence, with conditioning factors controlling the approximation. From an RL perspective, this suggests an inductive bias for value estimation: if policy values approximately satisfy low-order recurrences, encouraging this property may reduce the effective complexity of critic learning without excluding accurate value predictions. This motivation is consistent with RL theory connecting sample efficiency to suitable complexity measures of value-function classes under additional assumptions, such as the Bellman Eluder dimension [2]. We have not established that low Hankel rank controls these complexity measures; proving a sample-complexity benefit for Q-Hankeling remains an open question.

[2] Chi Jin, Qinghua Liu, and Sobhan Miryoosefi. Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithms. NeurIPS, 2021.

Q2: On critic and visitation effects

Our rollout diagnostics freeze the policy and critic at each checkpoint and collect fresh evaluation trajectories (Section 4.2 and Appendix D.3). This avoids policy changes within each evaluation, but both the critic and the trajectory distribution can change across checkpoints. As training approaches convergence, the policy and critic may stabilize, bringing the learning process closer to the fixed-policy setting considered in Section 4.3. This offers a possible connection between the analysis and late-training behavior, although a stable critic need not equal the true policy values, and the mixing assumptions remain additional conditions. These measurements alone therefore cannot separate structure in true policy values from approximation and visitation effects. Following the reviewer's suggestion, we conducted this control on Classical Control with the baseline DQN (five seeds). At each checkpoint, we compare the critic predictions with the discounted returns along greedy evaluation trajectories. Since policy and dynamics are deterministic, these returns equal QπQ^\pi exactly on trajectories that terminate. The three tasks of the paper have a constant reward at every step, so their returns have low Hankel rank under any policy, and we therefore consider an Acrobot variant with a dense reward (Table G below). There, the Monte Carlo estimates have low normalized Hankel rank throughout training, whereas the critic predictions at initialization do not. As training proceeds, the critic predictions closely match the Monte Carlo estimates and their rank decreases, remaining at least as large as the Monte Carlo rank on 284 of 285 trajectories after the first 5,000 steps. This is empirical evidence that, in this setting, the low-rank structure is present in the true policy values and not introduced by function approximation. The evidence is limited to DQN on small deterministic tasks, and extending the control to MuJoCo and Atari-100k remains open. We will include this control in the revised manuscript.

Table G. Monte Carlo control on Acrobot with a dense reward, using the baseline DQN and five seeds. Entries are medians over greedy rollouts that terminate. Ranks are effective Hankel ranks at 99.9% energy divided by the matrix dimension. Correlation is between the critic predictions and the Monte Carlo estimates along a rollout.

TrainingRolloutsCritic rankMonte Carlo rankCorrelation
Start100.4710.0220.772
25%750.1460.0770.998
50%750.1360.0750.999
75%750.1390.0830.999
100%300.1340.0950.999

Thank you for the reply. My questions have been addressed, and I've adjusted the score accordingly. I think the new experiments are valuable in understanding the cause of this phenomenon, and I encourage the authors to add more discussions on what properties of the MDP could be causing this.

Reviewer_XDDoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper leverages the geometric mixing properties of Markov chains under a stationary policy to prove that value sequences admit low-order approximate linear recurrences. Furthermore, building on Kronecker's theorem and Theorem 4.1, it establishes that this recurrence property yields an approximately low-rank structure in the corresponding conditional expected Hankel matrices. Motivated by these theoretical findings, the authors introduce Q-Hankeling, a lightweight regularization method. This method avoids explicit Hankel matrix construction and SVD by sampling consecutive short trajectory windows from replay buffer and applying a Huber loss to penalize order-rr linear recurrence residuals of critic value predictions, thereby introducing low temporal complexity as an inductive bias during critic learning to improve sample efficiency. Integrating Q-Hankeling into standard value-based and actor-critic baselines, extensive experiments demonstrate that the method improves sample efficiency and final policy quality across a range of tasks.

Critical strengths
  1. The motivation of the paper is well-justified and notably novel. The research motivation is grounded not only in empirical observations but also in supporting theoretical analysis. Compared with prior low-rank related work, the paper identifies a new object of study—low-rankness of value functions along the temporal axis of trajectories—thereby broadening the scope of low-rank research and turning the question of which axis low-rankness should be analyzed along into a question worth discussing.

  2. The proposed method enjoys broad adaptability (across different types of RL algorithms), wide applicability (across diverse task environments), and low computational cost, and thus has potentially significant impact from an application perspective.

Critical weaknesses
  1. Although the low-rankness of value sequences proposed by the authors is indeed novel, from its final implementation form, the method is essentially a penalty along the temporal axis. Regularization at the temporal level is not entirely unprecedented in prior work—for example, Thodoroff, Pierre, et al., "Temporal Regularization for Markov Decision Process," Advances in Neural Information Processing Systems 31 (2018)—yet this work is not mentioned in the paper.

  2. From the main experimental results in Table 1, the method yields negative improvements on several tasks (e.g., a 6% drop on Acrobot in Classical Control). Furthermore, the hyperparameter λ\lambda is overly sensitive to changes in task value scales. No automatic calibration scheme across tasks is provided, which increases deployment cost.

Questions for authors
  1. With respect to Weakness 1, the authors should further delineate the boundary between their method and existing temporal regularization approaches, and supplement the comparison or ablation studies with additional experiments to address whether the performance improvement originates from smoothing or adaptive smoothing;

  2. Whether a multi-step window is necessary? If the observed performance is comparable to that of single-step smoothing regularization (r=1), the significance of the temporal low-rank structure proposed in this paper would be considerably diminished;

  3. The paper acknowledges (D.7) that no equal-wall-clock control has been run, so the gains are not established at matched compute. Please report per-update FLOPs and wall-clock curves, and run an equal-wall-clock comparison (e.g., baseline with higher replay ratio) to quantify whether the sample-efficiency advantage survives the r=8 compute overhead; a cost–benefit curve over r=2,4,8 would also clarify whether the expensive settings are necessary.

4 comments

We thank the reviewer for their comments and valuable feedback, which are helpful in revising our manuscript.

W1: Comparison to Thodoroff et al. (2018)

We thank the reviewer for this highly relevant reference, which we will discuss in the revised manuscript. We agree that Q-Hankeling is a form of temporal regularization with precedents. Our contribution concerns the observed Hankel structure and the recurrence penalty it motivates. Three distinctions are relevant.

  • (i) Coefficients. The implemented smoothing methods in [1] use nonnegative weights summing to one. Q-Hankeling learns unrestricted coefficients, many of which are negative by the end of training.
  • (ii) Objective. The method in [1] modifies the bootstrap target. Q-Hankeling retains the base targets and adds a recurrence-residual penalty, learning coefficients jointly with the critic.
  • (iii) Theory. The analysis in [1] studies contraction and induced value-function bias. Ours connects Hankel rank to recurrences and investigates recurrence errors under mixing.

We will clarify these distinctions in the revised manuscript. Q1 discusses the new fixed-coefficient comparisons.

[1] Pierre Thodoroff, Audrey Durand, Joelle Pineau, and Doina Precup. Temporal Regularization for Markov Decision Process. NeurIPS, 2018.

W2: On the performance of Q-Hankeling

We acknowledge that Q-Hankeling does not improve performance in all tested environments, as reported in our manuscript (see Table 1, including the Acrobot decline). Nevertheless, it offers improvements in aggregate, including higher mean, median, and interquartile mean human-normalized scores on Atari-100k (Table 2). Appendix E.5.3 (Calibrating the penalty weight) provides guidance for calibrating λ\lambda using relative gradient magnitudes. This remains a heuristic rather than an automatic scheme across tasks. We will make this guidance and its limitations more prominent in the revision.

Q1: On the purpose of Q-Hankeling

We acknowledge that distinguishing Q-Hankeling empirically from existing temporal smoothing methods such as [1] was not an explicit focus of our original experiments. Our aim was to investigate whether a regularizer that encourages the observed low-rank Hankel structure could improve deep RL performance, rather than establish superiority over other temporal regularization methods. Our experiments show that adding Q-Hankeling improves aggregate performance, while the Hankel diagnostics provide empirical evidence that the penalty encourages the intended structure (Appendix E.4.3). To investigate the role of learning the coefficients, we compare against fixed smoothing and other fixed recurrences (Tables A and B in our response to Reviewer QoJW). Several fixed variants achieve higher point estimates. These comparisons do not yet isolate whether the gains arise from smoothing or from learning a richer recurrence. All variants use the regularization weight selected for the learned configuration, without independent retuning. We will include these results and clarify this open question in the revised manuscript.

Q2: Is a higher-order recurrence necessary?

Our current comparisons do not establish that higher-order recurrences are necessary. Their benefit may depend on the complexity of the value sequences. Classical Control already exhibits very low effective Hankel rank, so gains with r=1r=1 (Table B, response to Reviewer QoJW) are consistent with a low-order recurrence sufficing in these settings. However, fixed smoothing also performs well in the five-game Atari comparison (Table A, response to Reviewer QoJW), so this interpretation does not explain the full result. Whether higher orders help with richer temporal structure, and how to select rr from the observed structure, remain open questions. We will discuss these limitations and the new r=1r=1 results in the revised manuscript.

Q3: Computational cost

Our main comparisons use matched environment-interaction and gradient-update budgets. Q-Hankeling adds rr critic evaluations per replay sample, trading additional computation for the observed gains at a fixed interaction budget. We now report per-update costs on Atari-100k (Table E) and an equal-compute control on Classical Control (Table F).

Per-update cost. On Atari-100k, the paper's configuration (r=8r=8) uses 6.2× the FLOPs of the baseline per update and 1.87× its time. The cost is set by the order rr and not by whether the coefficients are learned. At both r=2r=2 and r=8r=8, learned and fixed coefficients have the same FLOPs and the same time per update to within 2%, so we report Table E with one row per order. Orders r=1r=1 and r=2r=2 use 2.2× and 2.8× the FLOPs and 1.62× and 1.67× the time. On MuJoCo, Appendix D.7 (Measured runtime, Table 7) reports runtime ratios of approximately 1.05–1.2× at r=2r=2 and 1.3–2.0× at r=8r=8.

Equal compute. On Classical Control, Q-Hankeling at r=2r=2 increases wall-clock time per run by 1.17–1.23×. We therefore trained the baseline with 1.25× and 1.5× gradient updates per environment step, which matches or exceeds this overhead. The additional updates do not reproduce the CartPole gain. On MountainCar the 1.5× baseline exceeds the learned variant and remains below the fixed ones (Table B, response to Reviewer QoJW), and the Acrobot decline persists for every variant. With five seeds these comparisons are not conclusive. Because of compute constraints we do not yet have an equal-wall-clock comparison on Atari-100k or MuJoCo, or return-versus-time curves. We will state this scope in the revision so that the reported gains are interpreted under the budgets actually matched in our experiments.

Table E. Cost per gradient update on Atari-100k with DER, by recurrence order rr. The cost at a given order is the same for learned and fixed coefficients, so each order has one row. Times are from the runs on NVIDIA A100 GPUs. Training time is the wall-clock time of the whole training loop. Ratios to the baseline are in parentheses.

OrderFLOPs per updateTime per update (ms)Training time (min)
Baseline (no penalty)1.07×10101.07\times10^{10} (1.00×)9.34 (1.00×)35.2 (1.00×)
r=1r=12.35×10102.35\times10^{10} (2.20×)15.09 (1.62×)53.2 (1.51×)
r=2r=22.98×10102.98\times10^{10} (2.78×)15.58 (1.67×)54.8 (1.56×)
r=8r=8 (paper setting)6.64×10106.64\times10^{10} (6.20×)17.50 (1.87×)61.4 (1.74×)

Table F. Equal-compute control on Classical Control with DQN and five seeds. The two control rows train the baseline with more gradient updates per environment step. The baseline row reports final return, mean (SE). The remaining rows report percentage changes relative to baseline, as in Table B of our response to Reviewer QoJW.

VariantCartPoleMountainCarAcrobot
Baseline407.0 (77.3)-131.8 (6.7)-73.6 (1.1)
Learned cc, r=2r=2 (paper setting)+8%+4%-6%
Baseline with 1.25× gradient updates-19%+2%-4%
Baseline with 1.5× gradient updates-15%+9%-1%
Reviewer_XDDoOpenReview ↗

Official Review of Submission49949 by Reviewer XDDo

Table A and Table B show that, in most cases, the method proposed in this paper performs worse than the r = 1 smoothing regularization method, which makes it difficult for me to understand why accounting for higher-order recurrences fails to yield effective performance gains. Does this support the value of the low-rank Hankel regularization?

We thank the reviewer for the quick follow-up. Our goal was to investigate whether the observed low-rank Hankel structure could improve deep RL learning, without claiming the best temporal regularization. We agree that r=1r=1 smoothing performs well on the five Atari games and Classical Control suite reported in Tables A and B.

  • Smoothing is itself a special case of the recurrence penalty ∥Qt−∑i=1rciQt−i∥22\lVert Q_t-\sum_{i=1}^{r}c_iQ_{t-i}\rVert_2^2, with r=1r=1 and c1=1c_1=1. Its exact recurrence gives a constant sequence of Hankel rank at most one; Theorem 4.1 connects approximate recurrences to approximate low rank. Its success is therefore consistent with the structural motivation.

  • The new diagnostics help explain why low order may suffice here. Baseline Atari window ranks are close to one at 95% energy and about one to three at 99% (Table H); Classical Control rollout ranks are around one or two. The learned r=8r=8 coefficients also concentrate on early lags: by the end of training, the first lag carries about 80% of the coefficient weight, and each coefficient at lags three to eight has magnitude below 0.1. This suggests limited use of the additional flexibility, although effective rank alone does not determine the best order.

Table H. Per-game diagnostics and performance. Baseline ranks use n×17n\times17 replay-window matrices (n≤32n\leq32), averaged over ticks in the final 5,000 environment steps and four seeds (Figure 21 of the manuscript). Learned c1c_1: final median over five seeds, initialized at 2.01. Performance columns: percentage changes in mean HNS relative to baseline, five seeds.

GameRank at 95% / 99% / 99.9%Learned c1c_1Learned cc, r=8r=8Smoothing, r=1r=1Linear, r=2r=2Quadratic, r=3r=3
Boxing1.0 / 2.3 / 9.90.89+9%+10%-20%+36%
Breakout1.1 / 3.1 / 8.01.12+46%+11%-18%-4%
DemonAttack1.0 / 1.6 / 6.51.26-24%-12%-15%-22%
Jamesbond1.1 / 2.5 / 7.11.28-25%-9%0%-13%
Krull1.0 / 1.1 / 4.61.13+26%+43%+49%+47%
  • Smoothing and learned r=8r=8 regularization also reduce replay-window rank similarly (Table I): raw ranks fall from 2.2 to 1.0, and centered ranks from 6.9 to 3.5 and 3.7, respectively. Final rollout ranks remain similar to the baseline, locating the observed reduction in training-time replay windows.

Table I. Final effective ranks on the five Atari games (DER, λ=2\lambda=2; median over 25 run summaries per variant). Window ranks: 99% energy, matrix construction and within-run averaging as in Table H; centering subtracts the matrix mean. Rollout ranks: 99.9% energy, normalized by attainable rank and averaged over chunks within each run (Figure 25 of the manuscript).

VariantWindow rank, rawWindow rank, centeredRollout Hankel rank
Baseline2.26.90.433
Learned cc, r=8r=81.03.70.459
Smoothing, r=1r=11.03.50.439
Linear, r=2r=21.03.50.459
Quadratic, r=3r=31.03.50.434
Fixed exponential weights, r=8r=81.04.50.378

These results support recurrence regularization, with task-dependent benefits from higher orders: the learned r=8r=8 recurrence outperforms smoothing on Breakout, while the quadratic r=3r=3 recurrence does so on Boxing. Thus, first-order smoothing is not uniformly preferable, although these comparisons do not establish a consistent advantage for higher orders. More complex environments with richer temporal value structure may require higher orders rr, a possibility that remains to be investigated. We will report these additional diagnostics in the revised manuscript and further investigate when richer temporal value structure benefits from higher recurrence orders. We thank the reviewer for this important observation.

Reviewer_QoJWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether critic value sequences along RL trajectories exhibit low effective Hankel rank. It connects approximate Hankel rank to linear recurrences and establishes low-rank approximation results for conditional expected Hankel matrices under mixing assumptions. Motivated by these results, Q-Hankeling jointly learns recurrence coefficients and regularizes critic outputs on replay windows. Experiments cover Classical Control, MuJoCo, and Atari-100k. Reported improvements include mean Atari HNS increasing from 0.463 to 0.530 and higher final returns in seven of eight MuJoCo task–algorithm combinations.

Critical strengths
  1. The paper offers an interesting perspective on temporal structure in scalar value estimates. The recurrence formulation provides a simple regularizer that avoids explicit Hankel construction and SVD.
  2. The method integrates with several established RL algorithms and shows promising gains across multiple benchmarks. The shared Atari configuration and matched interaction/update budgets strengthen the empirical evidence.
Critical weaknesses
  1. The connection between the central hypothesis and the practical regularizer remains incomplete. The mixing theory concerns conditional expectations, while training penalizes sampled trajectories. Moreover, Fig. 25 shows substantially higher effective ranks after centering and similar final rollout ranks with and without regularization. Although the paper acknowledges these limitations and reports reduced replay-window ranks, the evidence does not yet establish why a shared low-order recurrence is an appropriate structural prior.
  2. The experiments do not sufficiently isolate the benefit of learning recurrence coefficients. Comparisons against fixed first-order smoothing and fixed second-order extrapolation are particularly important: these simpler penalties could explain the improvements through temporal smoothing alone. Without these controls, the distinctive algorithmic contribution remains uncertain.
Questions for authors
  1. Evaluation request: Can you compare learned coefficients against fixed first-order smoothing and second-order extrapolation under matched tuning budgets? This would materially affect my assessment of the method’s contribution.
  2. Clarification: How do you reconcile the central low-rank hypothesis with the centered spectra and largely unchanged final rollout ranks in Fig. 25? What existing evidence connects the replay-window structural changes to performance gains?
  3. Clarification: Is the coefficient-sum constraint maintained throughout training, or only at initialization? How does this affect the connection to the theoretical analysis?
4 comments

We thank the reviewer for their comments and valuable feedback, which are helpful in revising our manuscript.

W1

We agree with the reviewer that our findings are far from providing a complete picture of how to exploit the observed low-rank Hankel structure in practice. Nevertheless, we believe they provide insightful guidance towards this goal.

Insight 1: From low-rank structure to a practical regularizer. Theorem 4.1 shows an equivalence between approximate low-rank Hankel structure and low-order linear recurrences under some assumptions. This motivated penalizing recurrence residuals of the form ∥Qθ(st,at)−∑i=1rciQθ(st−i,at−i)∥22\left\lVert Q_\theta(s_t,a_t)-\sum_{i=1}^{r}c_i Q_\theta(s_{t-i},a_{t-i})\right\rVert_2^2, using a Huber loss in the implementation. This efficiently encourages low-rank Hankel structure without constructing Hankel matrices or computing their singular values. The diagnostics in Appendix E.4.3 provide empirical evidence that the penalty encourages this structure alongside performance gains. We discuss their interpretation in Q2. Establishing theoretically when this penalty induces low-rank Hankel structure during critic learning, and when that structure leads to performance gains, remains an important open question.

Insight 2: A possible explanation for the observed structure. Theorem 4.2 in Section 4.3 investigates one possible mechanism in the idealized setting of a fixed stationary policy π\pi and exact policy values. The theorem shows that mixing permits a small conditional recurrence bias for optimally chosen coefficients satisfying ∑i=1rciπ=1\sum_{i=1}^r c_i^\pi=1, while stochastic fluctuations can remain. The condition ∑i=1rci=1\sum_{i=1}^r c_i=1 preserves constant sequences and is an assumption of the analysis. Interestingly, the learned coefficients remain close to this condition even from random initializations that violate it. We report the evidence in Q3.

W2: What our experiments isolate

Motivated by the observed low-rank Hankel structure, our main experiments compare various base deep RL algorithms with and without Q-Hankeling to evaluate the benefit of the proposed regularizer. These comparisons use learned recurrence coefficients and matched environment-interaction and gradient-update budgets. The reported gains (see Tables 1 and 2 of the paper) demonstrate that adding Q-Hankeling can improve performance across the evaluated settings, although the benefits are task dependent. Q-Hankeling also encourages low-rank structure on replay windows by the end of training (see Appendix E.4.3, Hankel diagnostics). These comparisons do not establish whether learned coefficients are preferable to fixed ones, or how best to learn them. We investigate these questions through the additional ablations in Q1.

Q1: Fixed smoothing vs. Learnable Coefficients

We now compare learned coefficients with fixed smoothing, linear, quadratic, and exponential recurrences (Tables A and B below). The smoothing, linear, and quadratic variants use c=(1)c=(1), c=(2,−1)c=(2,-1), and c=(3,−3,1)c=(3,-3,1), respectively. These fixed recurrences approximate patterns observed in the learned coefficients. The results are not conclusive as to whether learned coefficients are preferable to fixed ones. Several fixed recurrences achieve higher point estimates. The reported Atari intervals compare each variant with the baseline and do not directly quantify the uncertainty in the learned-versus-fixed comparison.

Table A. Fixed versus learned coefficients on five Atari-100k games (Boxing, Breakout, DemonAttack, Jamesbond, Krull), using DER, λ=2\lambda=2, and five seeds. HNS denotes human-normalized score. SE denotes standard error. Change is the percentage difference in mean HNS relative to the baseline. Every regularized variant uses the weight selected for the learned configuration, without independent retuning. Variants differ in recurrence order and computational cost.

VariantMean HNS (SE)ChangeDifference to baseline [±2 SE]
Baseline1.243 (0.067)——
Learned cc, r=8r=8 (paper setting)1.321 (0.086)+6%+0.078 [-0.139, +0.296]
Smoothing, r=1r=11.416 (0.076)+14%+0.173 [-0.028, +0.375]
Linear, r=2r=21.358 (0.072)+9%+0.116 [-0.081, +0.312]
Quadratic, r=3r=31.447 (0.104)+16%+0.205 [-0.042, +0.451]
Fixed exponential weights, r=8r=81.190 (0.060)-4%-0.053 [-0.233, +0.127]

Table B. Classical Control with DQN and five seeds. Regularized variants use the paper's task-specific λ\lambda, selected for the learned configuration, without independent retuning. Learned and fixed variants are compared at the same orders r=1r=1 and r=2r=2. The baseline row reports final return, mean (SE). The remaining rows report percentage changes relative to baseline, following the paper's convention.

VariantCartPoleMountainCarAcrobot
Baseline407.0 (77.3)-131.8 (6.7)-73.6 (1.1)
Learned cc, r=2r=2 (paper setting)+8%+4%-6%
Learned cc, r=1r=1+23%+7%-8%
Smoothing, r=1r=1+19%+13%-4%
Linear, r=2r=2+23%+16%-8%
Quadratic, r=3r=3+6%-17%-5%

Q2: On the Hankel diagnostics

As discussed in W1, these diagnostics test whether Q-Hankeling encourages the low-rank Hankel structure observed when using deep RL baselines. The diagnostics in Appendix E.4.3 (Hankel diagnostics) show that Q-Hankeling lowers effective rank on replay windows during training, including after centering. The higher centered ranks indicate that the mean value level accounts for part of the observed low-rank structure. Final greedy-rollout ranks are similar between the baseline and Q-Hankeling, despite the observed performance differences. These findings support an effect on the structure of values during learning, although they do not establish that rank reduction causes the gains.

Q3: Coefficient sum

We enforce ∑i=1rci=1\sum_{i=1}^r c_i=1 only at initialization. The coefficients are subsequently learned without projection or a sum-preserving parameterization. Surprisingly, their sum remains close to one across training checkpoints in all three suites (Table C). This behavior also occurs from random initializations. Starting from sums between 5.3 and 15.1, 137 of 140 Classical Control runs finish within 0.01 of one, with the remaining three ending at approximately 0.97, 0.98, and 1.06 (Table D). Thus, proximity to the theoretical condition is not solely inherited from initialization. We will revise the manuscript to discuss these coefficient dynamics and the fixed-coefficient comparisons in Q1.

Table C. Sum of learned recurrence coefficients, 1⊤c\mathbf{1}^\top c: mean ± standard deviation across runs at each training checkpoint. The recurrence order rr follows Table 1 of the paper. These additional runs initialize c=(1+1/γ,−1/γ,0,…,0)c=(1+1/\gamma,-1/\gamma,0,\ldots,0), rather than using the manuscript's random initialization subject to ∑ici=1\sum_i c_i=1. Both initializations sum to one, with no constraint enforced during training.

SuiteOrder rrInit25%50%75%100%
Classical Control (DQN)21.0000 ± 0.00000.9991 ± 0.00070.9979 ± 0.00190.9971 ± 0.00210.9969 ± 0.0024
MuJoCo (SAC and TD3)2 or 81.0000 ± 0.00000.9993 ± 0.00200.9998 ± 0.00351.0012 ± 0.00371.0010 ± 0.0037
Atari-100k (DER)81.0000 ± 0.00000.9960 ± 0.00520.9967 ± 0.00360.9969 ± 0.00430.9969 ± 0.0042

Runs averaged. Classical Control: 15 runs (3 environments × 5 seeds). MuJoCo: 37 runs across 5 environments and 8 task–algorithm pairs, with 5 seeds per pair except Swimmer with SAC (2 seeds). 20 runs use r=2r=2 and 17 use r=8r=8. Atari-100k: 130 runs (26 games × 5 seeds).

Table D. Random initialization of recurrence coefficients on Classical Control (DQN, r=2r=2, with the task-specific λ\lambda from Table 1 of the paper). Coefficient sums are mean ± standard deviation across runs at each training checkpoint. Four random starts drawn from U(0,10)2U(0,10)^2 have sums 5.3, 9.1, 10.3, and 15.1. No sum constraint is enforced.

TaskRunsInit25%50%75%100%
CartPole4 × 15 = 609.96 ± 3.500.9915 ± 0.01270.9939 ± 0.01460.9974 ± 0.00640.9984 ± 0.0046
MountainCar4 × 15 = 609.96 ± 3.501.0000 ± 0.00010.9997 ± 0.00060.9995 ± 0.00150.9997 ± 0.0076
Acrobot4 × 5 = 209.96 ± 3.560.9993 ± 0.00090.9983 ± 0.00140.9954 ± 0.00250.9943 ± 0.0023

Runs are four random starts × the indicated number of seeds.

The new comparisons have addressed my main concern, though the advantage of learned coefficients over fixed smoothing has remained unclear. Given the gains across tasks, I have raised my rating from 2 to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.