- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper, the authors study representation learning in reinforcement learning based on asymmetric Laplacian operators. Many existing Laplacian representation-learning approaches assume symmetric (self-adjoint) operators, whereas asymmetric Laplacians may have complex eigenvalues and non-orthogonal eigenfunctions, potentially making direct extensions of existing approaches challenging. The authors propose to address this problem through a two-phase approach. First, a neural network learns real-valued feature vectors whose span is approximately invariant under the forward transition operator. Second, reduced transition matrices are estimated for different horizons, and the eigenvectors of a weighted combination of these matrices are used to construct approximations of the asymmetric Laplacian eigenfunctions. In the experiments, the authors evaluate the proposed framework in simulated grid and drone environments.
Critical strengths
- The paper addresses an interesting and technically challenging problem. Learning representations that capture asymmetric properties of transition dynamics is not only practically relevant but also interesting in its own right.
- The proposed two-phase approach appears to be nontrivial, well motivated, and sensibly designed. The idea of avoiding the complexity of directly learning complex-valued eigenfunctions of asymmetric Laplacians could potentially find applications in other domains.
- The paper is well written. While I am not an expert in this particular domain, I was able to follow the main ideas reasonably well.
Critical weaknesses
-
While the motivation from the subspace iteration literature is reasonable, the paper does not seem to provide a rigorous theoretical analysis of the quality of the spectral approximation obtained through the proposed scheme, which encourages approximate invariance under the forward transition operator. This is not necessarily a serious limitation in itself and could be compensated for by extensive experiments and validation in real-world settings. However, the experiments are limited to simulated environments. Given the limited theoretical analysis and the absence of real-world experiments, I feel that there is considerable room to strengthen the contribution.
-
Since the asymmetric Laplacian is defined as L=I-P, its eigenfunctions are identical to those of the Markov transition operator P. Furthermore, the proposed learning scheme appears to operate directly on P. In this sense, I was wondering what additional advantage is gained by formulating the problem in terms of the asymmetric Laplacian rather than directly considering the spectral properties of the transition operator. In this respect, I think the paper would benefit from a more detailed discussion of how the proposed framework compares with existing approaches based on Markov or Koopman operators.
Minor comment
Some tables report numerical values to five decimal places, resulting in rather small font sizes. The authors could consider reducing the number of decimal places to improve readability.
Questions for authors
Please see Weaknesses section.
We thank the reviewer for their thorough reading of the paper and address their concerns below.
Weakness 1.
The application of asymmetric spectral methods to real-world settings is part of the motivation for this paper. Notably, the symmetry assumption in prior Laplacian methods can be problematic in real-world settings. Whilst evaluation on physical robots remains a future direction for this line of research, we find it important to follow the same evaluation procedures as prior symmetric Laplacian methods, such as GDO, GGDO, and ALLO. Specifically, one experimental limitation is that, to evaluate the quality of our learned spectral representations directly, we need environments where we can compute or estimate oracle eigenvectors.
There are two ways in which we can think about the quality of our representations: first, how accurately they represent the underlying spectrum; and second, how they can be used for downstream tasks. For the first, we follow the direction of related papers, including GGDO and ALLO, by evaluating how well the learned representations approximate oracle eigenvectors in environments for which we can obtain them.
That said, in light of the reviewer’s comment, we have included the application of ALR based hitting-time distance functions in simulated robotic control domains. We consider variations of Ant and Humanoid control tasks as start-and-goal tasks, where we use ALR distance functions for zero-shot goal tasks with policies from OGBench trained on a different domain than evaluated. Specifically, the distances are used to find intermediate waypoints for the pretrained agents to follow. The full details are provided in Section 4 and Appendix F.
In these experiments, we show clear improvements in using ALR hitting-time distance estimates to solve start to goal tasks compared with oracle RA-LapRep, an oracle version of the symmetric commute-time-based distance approach from Wang et al. [1]. Hitting time is the natural directed version of their commute-time distance, as such their comments on the challenge of estimating such as distance from the spectrum in directed settings is worth noting. Table 1 reports mean success rates of 54–96% for ALR across the six domain–controller combinations, compared with 0% for oracle RA-LapRep in this waypoint-planning setup. Here, RA-LapRep is given oracle eigenvectors (to ensure approximation isn't the reason they fail), whilst ALR distances are obtained from learned image representations. This comparison shows the downstream usefulness of learning spectral distances that retain the asymmetry of the environment.
Laplacian representations have already been found useful in reinforcement learning for distance estimation, bottleneck discovery, planning, and option discovery. We therefore consider this a non-trivial step in the right direction. We hope the reviewer will agree that these additional experiments strengthen the contribution by demonstrating the applicability of our method and making progress on the limitation identified in prior symmetric Laplacian distance methods.
Weakness 2.
Thank you for the suggestion. We have updated the manuscript to include a range of Koopman operator-learning baselines: Neural Power Iteration (NPI) [2], DPNet and its variation DPNet Relaxed [3], EncoderOps [4], and inexact subspace iteration (ISI) [5]. For ISI, we include both the Müller–Brown and AIB9 parameter settings, together with the released implementation and a corrected loss-scaling described in Section 4. We show these comparisons in Figure 3 of the updated manuscript. Specifically, in the four drone domains, we show that the ALR method obtains substantially higher mean cosine similarities for the first 20-50 eigenvectors associated with the lowest-magnitude Laplacian eigenvalues. We hope that these comparisons resolve the reviewer's minor concerns on the soundness of the proposed approach.
References:
[2] Berger, G. O., and Jungers, R. M. Data-driven Koopman mode approximation: A neural power iteration algorithm. arXiv:2608.26943, 2026.
[3] Kostic, V. R., Novelli, P., Grazzi, R., Lounici, K., and Pontil, M. Learning invariant representations of time-homogeneous stochastic dynamical systems. ICLR, 2024.
[4] Turri, G., Bonati, L., Zhu, K., Pontil, M., and Novelli, P. Self-Supervised Evolution Operator Learning for High-Dimensional Dynamical Systems. ICLR, 2026.
[5] Strahan, J., Guo, S. C., Lorpaiboon, C., Dinner, A. R., and Weare, J. Inexact iterative numerical linear algebra for neural network-based spectral estimation and rare-event prediction. The Journal of Chemical Physics, 159(1):014110, 2023.