acceptodds
Under review as a conference paper at ICLR 2027

Scalable Neural Quantum States with Modern Language Modeling

Abstract

Scaling neural representations to increasingly large quantum many-body systems remains a central challenge in quantum physics. Despite the success of Neural Quantum States (NQS), their dependence on costly Monte Carlo sampling and second-order optimizations fundamentally limits their scalability to large systems. Here, we introduce FastNQS, a transformer-based variational framework that combines autoregressive Born sampling, FlashAttention-accelerated wavefunction evaluation, and Muon optimization. Across benchmarks on the transverse-field Ising model, two-dimensional , , and random-bond Heisenberg models, FastNQS achieves competitive accuracy against Parallel RNN or ViT-based NQS baselines, while being significantly faster: an approximately speedup over ViT-based NQS in end-to-end runtime per step. The results highlight modern language modeling as a practical route toward scalable neural quantum states.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Neural Quantum States, Autoregressive Models, Transformers, Flash Attention, Variational Monte Carlo

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jV1eOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces FAST NQS, an autoregressive Transformer framework for approximating quantum many-body ground states. It combines a spatial positional encoder with KV-cached sampling, FlashAttention-based wavefunction evaluation, and Muon optimization. Experiments cover Ising, Heisenberg, frustrated J1-J2, and random-bond models, evaluating energy accuracy, correlations, runtime, and memory. The authors report competitive accuracy on several benchmarks and a 23.6× reduction in time per optimization step over a ViT-based baseline, although FAST NQS remains less accurate than the cited ViT reference on the two-dimensional frustrated benchmark.

Critical strengths

The strongest aspect is the practical reduction in computational cost. The reported 23.6× speedup per VMC step suggests that adopting modern Transformer tooling can substantially reduce the cost of NQS optimization, even though the advantage at matched accuracy remains unclear.

The validation also extends beyond energy estimates: the Ising experiments check long-range correlations against exact results. This provides evidence that the learned states capture meaningful physical behavior rather than merely achieving low energy.

Critical weaknesses

The methodological novelty is limited. The encoder appears to be the only novel component, but its connection to the quantum many-body task is not well motivated. It reads like an ad hoc modification to a general-purpose architecture, and the encoder ablation does not establish why this design is preferable to a simpler alternative. This makes the main architectural contribution difficult to justify.

The efficiency comparison does not establish faster ground-state estimation at matched accuracy. The headline speedup measures time per optimization step, yet FAST NQS remains less accurate than the cited ViT reference on the two-dimensional frustrated benchmark. Without energy-versus-runtime comparisons under comparable settings, the practical advantage central to the paper remains uncertain.

Questions for authors
  1. What property of the quantum many-body problem motivates the spatial encoder? Since its output is pooled into a single configuration-independent token, what evidence distinguishes its benefit from that of a learned constant vector or a parameter-matched decoder?
  2. Does FAST NQS reach a common energy-accuracy threshold faster than ViT-NQS under comparable experimental settings? The reported per-step speedup and the accuracy reference come from different comparisons, so it is unclear whether the speed advantage persists at matched accuracy.
Reviewer_PRFvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FAST NQS, a transformer-based framework designed to improve the scalability and computational efficiency of Neural Quantum States (NQS) for quantum many-body systems. The proposed approach combines autoregressive Born sampling with KV caching, FlashAttention-based wavefunction evaluation, Muon optimization, and a spatial positional encoder. The framework aims to address computational bottlenecks in variational Monte Carlo (VMC), including sampling, wavefunction evaluation, and parameter optimization. The authors evaluate FAST NQS on several quantum spin systems, including the transverse-field Ising model, two-dimensional Heisenberg models, frustrated J1-J2 systems, and random-bond Heisenberg models. The experimental results demonstrate substantial computational efficiency improvements while maintaining competitive variational accuracy. In particular, FAST NQS achieves a reported 23.6× reduction in end-to-end VMC iteration time compared with ViT-NQS on the selected two-dimensional J1-J2 benchmark, as well as a 394× reduction in peak forward-evaluation memory compared with minGRU under the reported configuration.

Critical strengths

The paper addresses an important practical challenge in neural quantum states: improving computational scalability without substantially sacrificing variational accuracy. The introduction provides a clear explanation of the limitations of existing sampling, attention, and optimization approaches, establishing a reasonable motivation for the proposed framework. The overall structure is logical, progressing from the VMC formulation to the proposed architecture and experimental evaluation. Figure 1 provides a useful overview of the framework, and the individual components are explained in an accessible manner. The explicit presentation of the main contributions also helps readers understand the intended advantages of FAST NQS.

Critical weaknesses

Although the framework is well motivated, several of its principal components, including KV caching, FlashAttention, and Muon optimization, are established techniques from modern deep learning. The paper's main contribution therefore appears to lie primarily in their integration and adaptation to NQS rather than in a fundamentally new learning algorithm. This is a reasonable and potentially valuable contribution, but the paper would benefit from a more explicit discussion of what is technically novel in the proposed integration. In particular, the role of the spatial positional encoder could be more clearly justified. Since the encoder produces configuration-independent global memory, an additional comparison with a simpler learned global-conditioning mechanism would help establish whether the proposed spatial mixing is essential. The existing architectural and optimizer comparisons are useful, but more systematic component-level ablations would make the individual contributions of KV caching, FlashAttention, and Muon easier to interpret. These additions would also strengthen the connection between the claimed architectural innovations and the observed efficiency improvements.

Questions for authors
  1. Could the authors further clarify the distinctive methodological contribution of FAST NQS? In particular, how much of the observed improvement can be attributed to the spatial positional encoder rather than the integration of existing KV caching, FlashAttention, and Muon techniques? A comparison with simpler global-conditioning alternatives and more explicit component-level ablations would be helpful.
  2. Could the authors provide a more detailed efficiency–accuracy comparison? Since the ViT-NQS reference achieves a lower variational energy on the frustrated J1-J2 benchmark, it would be useful to report wall-clock time or computational cost required to reach comparable target accuracy levels. Clarifying the optimizer configurations used in the runtime and accuracy comparisons would also improve interpretation.
  3. Could the authors clarify the experimental variability and memory measurement methodology? In particular, additional information about repeated training runs, statistical variability, and the measurement procedure behind the large forward-memory reduction relative to minGRU would help establish the robustness of the reported improvements.
Reviewer_fBhBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces FAST NQS, a transformer-based autoregressive Neural Quantum State framework designed to reduce the computational cost of Variational Monte Carlo. The method combines a spatial positional encoder with KV-cached autoregressive sampling, FlashAttention-accelerated wavefunction evaluation, and Muon optimization. It is evaluated on the one-dimensional transverse-field Ising model, the two-dimensional J1J_1 Heisenberg model, frustrated J1J_1-J2J_2 systems, and a random-bond Heisenberg model. Across these benchmarks, FAST NQS obtains useful variational accuracy while substantially reducing computational cost relative to selected autoregressive and ViT-based NQS baselines. The strongest reported efficiency result is a 23.6×23.6\times reduction in wall-clock time per VMC optimization step relative to a parameter-matched ViT-NQS implementation on the 10×1010\times10 frustrated J1J_1-J2J_2 benchmark, together with lower reported peak memory. The paper argues that adapting modern language-modeling infrastructure can provide a practical route toward more scalable NQS calculations.

Critical strengths

S1: FAST NQS demonstrates substantial practical efficiency gains across multiple stages of the VMC pipeline.
The strongest aspect of the paper is that the authors do not optimize only a single kernel or isolated component, but combine direct autoregressive sampling, KV caching, optimized attention kernels, and first-order optimization within a complete VMC pipeline. The reported wall-clock improvements are large, particularly the 23.6×23.6\times reduction in time per VMC step relative to the parameter-matched ViT-NQS implementation on the 10×1010\times10 frustrated J1J_1-J2J_2 benchmark. Since NQS calculations can be computationally expensive, improvements of this magnitude could be practically valuable even if many of the individual ingredients are adaptations of techniques developed elsewhere. This is the main reason I view the paper as potentially useful rather than as a clear rejection.

S2: The empirical evaluation covers several physically distinct and nontrivial regimes. The method is tested on an exactly solvable critical TFIM, the unfrustrated two-dimensional J1J_1 Heisenberg model, frustrated J1J_1-J2J_2 systems, and disordered random-bond Heisenberg instances. The paper also uses several kinds of reference information, including exact solutions, QMC, DMRG, Lanczos calculations, and existing NQS baselines. I particularly appreciated that the TFIM evaluation considers a long-range correlation observable in addition to energy, and that the random-bond experiment considers multiple disorder realizations rather than a single favorable instance. This breadth gives useful evidence that the computational optimizations are not restricted to one toy setting.

Critical weaknesses

W1: The current experiments do not yet establish that the headline efficiency gains translate into a comparably strong improvement in end-to-end accuracy-efficiency or practical scalability.
The main efficiency results are reported primarily as wall-clock time per VMC optimization step or forward-pass memory. However, architectures, samplers, and optimizers can require very different numbers of updates to reach the same variational quality. For this reason, I think the central comparison should be energy/error versus actual wall-clock time, or time required to reach a fixed accuracy threshold, rather than only cost per update. This matters particularly on the main two-dimensional frustrated benchmark: FAST NQS reaches approximately −0.4913-0.4913 energy per site, whereas the ViT reference is approximately −0.4977-0.4977, leaving an energy-density gap of about 0.00640.0064 (roughly 1.31.3% relative to the magnitude of the reference). The efficiency result is therefore interesting, but the paper does not yet show the corresponding time-to-accuracy trade-off.

Figure 4 also mixes two different ViT comparisons. The accuracy reference in panel (a) is taken from Rende et al. [1], whereas the parameter-matched ViT-NQS implementation timed in panels (b,c) is a different configuration and is optimized with Adam. Consequently, the current figure does not establish the accuracy-efficiency Pareto trade-off against the same baseline. The timing protocol for the parameter-matched ViT is also insufficiently specified. Since the ViT-NQS VMC step combines MCMC sampling, evaluation of Hamiltonian-connected configurations, backpropagation, and optimization, the 23.6×23.6\times result would be much easier to interpret with the ViT sampling parameters and a runtime breakdown. This particular 23.6×23.6\times measurement should not be interpreted as evidence for savings from avoiding SR, since the timed ViT baseline itself uses Adam.

I also find the reported 394×394\times forward-memory reduction in Figure 3 insufficiently specified. At L=32L=32 and B=2048B=2048, Table 7 gives dmodel=32+96=128d_{\mathrm{model}}=32+96=128, but the patch size is not reported. If, for example, p=2p=2, the sequence has T=(32/2)2=256T=(32/2)^2=256 tokens, and a single hidden-state tensor of shape B×T×dmodelB\times T\times d_{\mathrm{model}} already occupies 2048×256×128×2=134,217,7282048\times256\times128\times2=134{,}217{,}728 bytes, i.e. 128128 MiB in FP16/BF16, before other activations or parameters are counted. Even the smallest FAST NQS memory bars in Figure 3 appear to be of the same order as parameter storage alone. This does not necessarily mean that the reported 10.910.9 MiB value is wrong: the implementation may use chunking, micro-batching, an incremental-memory convention, or another measurement procedure. However, the definition of "peak memory" and the execution protocol need to be made explicit before the 394×394\times headline can be interpreted.

Finally, the experiments do not yet convincingly demonstrate that these efficiencies expand the physically accessible regime. The principal frustrated two-dimensional J1J_1-J2J_2 experiment remains at 10×1010\times10, and Appendix A shows visibly worse final error for larger Majumdar-Ghosh chains while the model width and depth (for N≥20N\geq20) remain fixed. The latter may reflect a capacity choice rather than a fundamental limitation of FAST NQS, but it further motivates separating computational throughput from accuracy scaling. Overall, I find the efficiency improvements promising, but the current evidence falls short of demonstrating a new end-to-end scalability frontier that the title and abstract seem to imply.

W2: The current ablations do not sufficiently isolate or explain several components presented as central to FAST NQS.
The spatial encoder is the clearest example. Appendix E states that the G2G^2 encoder representations are mean-pooled into a single configuration-independent memory token before decoder cross-attention. If the key/value sequence has length one, then the attention weight is exactly softmax⁡(qtk⊤/d)=1\operatorname{softmax}(q_tk^\top/\sqrt{d})=1 for every decoder position. Therefore the decoder cross-attention output is independent of the query and key, the corresponding query/key projection matrices receive zero gradient through this operation, and each position/sample receives the same value-projected conditioning vector within a given decoder layer. The encoder may still be useful as an overparameterized way to construct this vector, and Figure 4(a) suggests that adding it improves optimization or final energy, but its functional role appears substantially simpler than the paper's interpretation of dynamically accessing a "global representation of lattice geometry." A more diagnostic ablation would replace the encoder and single-token cross-attention with directly learned per-layer conditioning vectors. If this performs similarly, the claimed benefit would be better understood as a parameterization/optimization effect rather than spatial cross-attention.

The evidence for Muon is also too limited to support a strong optimizer-level conclusion. The comparison uses one fixed initialization, one "tuned" Muon learning rate, and an Adam learning-rate grid. All Adam settings from 10−410^{-4} to 10−310^{-3} finish extremely close to one another around −0.45-0.45, which is compatible with the optimizer becoming trapped in the same poor regime rather than with a simple learning-rate mismatch. The reported 7.5×7.5\times error reduction should therefore be interpreted as the improvement over the best Adam configuration in this particular sweep, rather than as robust evidence that Muon is generally superior for NQS optimization. Multiple seeds, complete Adam/AdamW settings, and, ideally, comparison with an NQS-specific alternative such as PWO, minSR, or SR would substantially strengthen this part of the contribution.

Additional observations (not central to my current rating).

  • The random-bond comparison in Figure 5(b) uses different optimization/sample budgets for the two methods: Appendix D trains the ViT-NQS baseline for 60006000 steps with 10241024 samples per step, whereas Table 9 gives 10,00010{,}000 steps and batch size 40964096 for FAST NQS. This does not invalidate the reported V-scores, but it should be made clear that this is not a matched-compute comparison.

  • Table 1 largely reports FAST NQS energies after D4D_4 fine-tuning, while the L=32L=32 result is not D4D_4 fine-tuned. Since the symmetry-averaged stage evaluates eight related configurations and incurs additional optimization cost, it would help to flag this difference directly in Table 1 and distinguish the accuracy protocol from the efficiency measurements.

  • There are several reproducibility details that should be clarified: the main text defines the TFIM experiment with J=1J=1 whereas Appendix F states J=−1J=-1; it is unclear whether the FAST NQS Heisenberg experiments enforce the Stotz=0S^z_{\mathrm{tot}}=0 sector during autoregressive sampling; and Eq. 12 symmetrizes probabilities under D4D_4 without specifying how the sign/phase is treated when the phase head is active.

  • The novelty discussion could be calibrated more carefully against prior autoregressive transformer NQS work. Transformer Quantum State [2] predates FAST NQS, and Xu et al. [3] explicitly discuss cache-centric/KV-cache optimization for transformer-based NQS sampling. These works do not appear to duplicate the particular combination of KV caching, FlashAttention, Muon, and the present spin-lattice architecture, but they make the contribution better described as a new integrated systems recipe than as the first application of modern transformer inference ideas to NQS.

References

[1] Rende et al., A Simple Linear Algebra Identity to Optimize Large-Scale Neural Network Quantum States, Communications Physics, 2024.

[2] Zhang et al., Transformer Quantum State: A Multipurpose Model for Quantum Many-Body Problems, Physical Review B, 2023.

[3] Xu et al., Large-scale Neural Network Quantum States for ab initio Quantum Chemistry Simulations on Fugaku, IEEE ISPA, 2025.

Questions for authors

Q1: Can the authors provide an apples-to-apples time-to-accuracy comparison against the same ViT-NQS implementation used for the timing experiment?
In particular, what final variational energy does the parameter-matched ViT from Figure 4(b,c) reach, and can you report energy/error versus wall-clock time for that model and FAST NQS? A result showing that FAST NQS reaches comparable accuracy substantially faster would materially strengthen my assessment.

Q2: What exact sampling configuration is used for the timed ViT-NQS baseline, and where does the 23.6×23.6\times runtime gap come from?
Please specify the number of chains, MCMC sweeps/proposals, discarded samples, and samples per VMC step used in Figure 4(b,c). A breakdown of wall-clock time into sampling, Hamiltonian-connected wavefunction/local-energy evaluation, backward pass, and optimization would make the comparison substantially easier to interpret.

Q3: How exactly is the 10.910.9 MiB FAST NQS peak-memory value in Figure 3 measured?
Please report the patch size, numerical precision, whether the full B=2048B=2048 batch is processed simultaneously or chunked/micro-batched, and whether the metric denotes total allocated GPU memory, incremental memory above model parameters, post-forward allocated memory, or another quantity. The reported value appears smaller than a single full-batch hidden-state tensor under the architecture described in the appendix.

Q4: Is the decoder cross-attention indeed given a single mean-pooled encoder token, and how does this differ functionally from a learned per-layer conditioning vector?
If Appendix E describes the implementation exactly, the single-key softmax is always one and the cross-attention query/key projections cannot influence the output. Could you clarify whether I am missing an implementation detail and, ideally, compare the current encoder against directly learned per-layer conditioning vectors?

Q5: How robust is the Muon advantage?
Could you report results across multiple seeds and the complete Adam/AdamW optimization settings, including schedule, warmup (if any), β\beta values, ϵ\epsilon, weight decay, and any other relevant choices? If feasible, a comparison with PWO [4], minSR [5], or SR [6] under a matched training/compute budget would also help determine whether the improvement is specific to comparison against Adam or generalizes to established NQS optimizers.

Q6: Could the authors clarify the remaining implementation/symmetry details relevant to reproducing the experiments?
Which TFIM coupling was actually used, J=1J=1 or J=−1J=-1? Do the FAST NQS Heisenberg runs constrain sampling to Stotz=0S^z_{\mathrm{tot}}=0, and if so how is this implemented for patch tokens? Finally, which experiments use the phase head, and how is the sign/phase handled during the D4D_4 probability-averaging fine-tuning?

The answers to Q1-Q4 are the most important to my overall assessment, because they directly determine how I interpret the paper's two headline efficiency results and the claimed role of the spatial encoder.

References

[4] Duque et al., One More Time: Revisiting Neural Quantum States from a Reinforcement Learning Perspective, arXiv, 2026.

[5] Chen et al., Empowering Deep Neural Quantum States through Efficient Optimization, Nature Physics, 2024.

[6] Sorella, Generalized Lanczos Algorithm for Variational Quantum Monte Carlo, Physical Review B, 2001.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.