acceptodds
Under review as a conference paper at ICLR 2027

F2: Offline Reinforcement Learning for Hamiltonian Simulation via Free-Fermionic Subroutine Compilation

Abstract

Compiling shallow and accurate quantum circuits for Hamiltonian simulation is challenging due to hardware constraints and the combinatorial complexity of minimizing gate count and circuit depth. Existing compilers rely on hand-engineered heuristics and cannot adapt to input-dependent structure, limiting optimization opportunities. We introduce F2, an offline reinforcement learning framework that exploits free-fermionic structure for quantum simulation. Across benchmarks spanning lattice models, protein fragments, and crystalline materials (12–222 qubits), F2—using a single trained model—reduces count by 28.8% on average relative to strong industry and academic baselines (IBM Qiskit, Google Cirq/OpenFermion, Berkeley BQSKit), while maintaining errors on the order of . These results demonstrate that aligning reinforcement learning with algebraic structure can substantially improve scalable quantum circuit synthesis.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Quantum Computing, Reinforcement Learning, Drug Discovery, Material Discovery

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VVfWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces F2, an offline reinforcement learning framework which reduces compilation cost in quantum simulation by targeting free-fermionic structure. The method applies trajectory-reversal data generation to efficiently generate successful traces without environment interaction, and designs geometric potential function and regularizer to shape the training. The experiments show an average reduction of 28.8% compared with industrial baselines on a benchmark spanning Hamiltonians of different families and qubit ranges.

Critical strengths

The overall problem formulation and framework design is natural. Techniques such as reversal data generation and potential reward shaping are established methods but well applied here.

Reducing compilation cost for Hamiltonian simulation is an important research problem in quantum computing. The framework achieves relatively strong empirical performance. It is interesting that optimizing the classcially tractable part of Free Fermionic structure can produce such improvements.

Critical weaknesses

I have concerns about the scalability for system size of F2. It is confusing to say "U∈SO(2N)U\in SO(2N) where N is the number of free-fermionic modes" in Section 2.1, but "the trained policy is reused across all benchmark instances and domains, without per-instance or per-family retraining" in Section 3.1, because the benchmark covers instances of different qubit sizes. The authors should have made windowing for larger systems explicit in the main text. The windowing strategy shows F2's limitation in directly handling systems with large qubit size. The experimental results further amplifies this concern, as 2D Hamiltonians are only benchmarked at qubit number = 12, while omitted at 72 or 144.

Other experiments clarification. (1) The authors specify that baselines such as IBM-base uses first-order trotter. Does F2 use first-order trotter as well? (2) The result table only includes RzR_z gate count. The authors should also present the simulation error for each case as well, as error budget significantly influences the gate count. (3) The experimental setup didn't specify the coupling values, boundary conditions, evolution time, and other parameters for the benchmark Hamiltonians. (4) There's a small peak of probability mass at 10−210^{-2}, suggesting the method does not reliably achieves error around 10−710^{-7} as claimed. What happens when a compilation exceeds the error threshold and is rejected? How to obtain a valid compilation then? (5) The paper reports three to four orders of magnitude of compilation time overhead despite running on one A100, which limits its practicality.

Questions for authors
  1. Could you conduct experiments for 2D Hamiltonians with qubit number 72 and 144?
  2. Could you address the concerns about experiments in the second part of weaknesses?
  3. Some references seem to be considerably wrong. For example, Practical and Efficient Quantum Circuit Synthesis and Transpiling with Reinforcement Learning is by Kremer et al.
Reviewer_cEm8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

F2 uses offline reinforcement learning to compile Hamiltonian-simulation Trotter steps. It separates free-fermionic and interacting terms, represents the former with compact Majorana/SO(2N) matrices, and learns sequences of Pauli-string exponentials that reduce a target unitary to the identity. Training uses 10,000 successful reverse-generated trajectories containing 1,502,443 frames. A dual-tower value network combines Monte Carlo targets, reward shaping, geometric regularization, and TD refinement; inference greedily scores successor states. A fixed 12-mode model is reused across windows, while Rustiq handles interacting and cross-window terms. Across 20 benchmarks with 12–222 qubits, the reported mean RZ-count reduction is 28.8% relative to the best alternative on each benchmark. Depth is best on 5 of 20 cases, and sampled unitary tests report small compilation errors. (Sections 2–3; Tables 1–2; Figure 2; Appendix C.)

Critical strengths
  1. The free-fermionic representation makes the synthesis state tractable and its transitions deterministic. Exact trajectory reversal then supplies successful training examples without solving each synthesis problem first. This is a sensible way to adapt offline learning to the algebraic structure of the task. (Sections 2.1–2.3; Equations 1–3.)
  2. The RZ-count improvements are measured across a varied set of benchmarks without retraining for each family. Reporting the weaker depth results alongside the favorable gate-count results helps establish the actual trade-off. (Tables 1–2; Section 3.2.)
Critical weaknesses
  1. The experiments do not separate the benefit of value learning from the decomposition, action vocabulary, and windowed synthesis. Figure 4 compares action embeddings through value loss, and Figure 2b evaluates Monte Carlo critics before final TD refinement. Neither provides a matched non-learning synthesis baseline. (Sections 2.3–3.2; Appendix E.)
  2. The practical scope needs clearer qualification. The 222-qubit result comes from repeated use of a 12-mode model, compilation is much slower than the listed baselines, and Section 2.1 and Appendix F state different termination criteria. These details affect both the scalability claim and confidence in the reported fidelity. (Appendices C and F.)
Questions for authors
  1. Which stopping rule produced the reported results? Section 2.1 uses the absolute normalized trace, which also accepts S = −I, whereas Appendix F uses normalized chordal distance. Please explain how false termination is excluded or detected and how the test relates to the geometric potential. (Section 2.1; Appendix F.)
  2. For the 72-, 144-, and 222-qubit benchmarks, what fraction of the Hamiltonian and final circuit cost is handled by F2 rather than Rustiq? This would make the extent of fixed-window reuse clear. (Appendix C; Tables 1–2.)
  3. What deployment setting justifies the compilation overhead? Table 4 reports 10.7–23.7 s at 12 qubits and 141–295 s at 144 qubits on GPU, versus less than 1.3 s for the listed CPU library baselines. Please relate these costs to the RZ savings and include the 222-qubit case if available. (Appendix F.)
  4. How well does the −1-per-action objective track native-gate cost? Depth is 1.2–2.2 times the best baseline on seven benchmarks. Two-qubit gate counts or a cost-aware objective ablation would help interpret the hardware relevance. (Section 2.1; Tables 1–2.)
  5. Can a geometric greedy or search policy be evaluated with the same actions and 12-mode partition? Task-level ablations should also isolate shaping, geometric regularization, and TD refinement. In Figure 2b, shaping alone raises median error from 1.34 × 10⁻⁴ to 6.30 × 10⁻⁴, while the combined regularizer lowers it to 6.17 × 10⁻⁶. What explains this interaction? (Figure 2b; Appendix E.)
Reviewer_cnaUOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an offline reinforcement learning framework for quantum circuit compilation in Hamiltonian simulation. The method exploits the algebraic structure of free-fermionic systems to formulate circuit synthesis as a sequential decision-making problem with a compact state representation. The authors generate successful training trajectories by sampling backward from the identity and introduce a geometric regularizer to improve value function learning. During inference, the learned value function guides greedy circuit synthesis. I have several concerns about the method and the experimental evidence. it is unclear how much of the improvement comes from reinforcement learning rather than the free-fermionic decomposition and the overall compilation pipeline. I also find the scalability claims somewhat overstated, given the fixed-size windowing approach. The choice of value-learning targets and the substantial compilation overhead raise further questions about the method's effectiveness and practical value.

Critical strengths

S1. The paper addresses a practically relevant problem. Quantum circuit compilation directly affects the resources required to execute quantum algorithms. Improving circuit efficiency for Hamiltonian simulation is a meaningful objective, and the paper has a clear application motivation. S2. The method makes good use of the structure of free-fermionic systems. By using the compact Majorana representation, the authors avoid working directly with exponentially large unitary matrices. Restricting the learning problem to this structured space makes the problem considerably more manageable and is a reasonable design choice. S3. The backward trajectory generation strategy is interesting. The authors exploit the reversibility of quantum operations to generate trajectories that are guaranteed to reach the target state. This provides a practical way to obtain successful training examples without relying on online exploration, which is particularly useful in problems with sparse rewards.

Critical weaknesses

W1. The contribution of reinforcement learning is unclear. It is difficult to determine how much of F2's improvement comes from reinforcement learning rather than the free-fermionic decomposition and specialized compilation pipeline. Although several compilers are evaluated, the paper lacks a controlled comparison with a non-learning method using the same partitioning, windowing, and post-processing. Such a baseline is important for establishing the value of the learned policy. W2. The scalability claims are somewhat overstated. While F2 is evaluated on systems with up to 222 qubits, the network itself only operates on fixed 12-mode windows. Larger systems are handled by applying the model independently to each window, with cross-window terms compiled by Rustiq. This demonstrates the scalability of the overall pipeline rather than the learned model. The paper also does not analyze how windowing affects circuit quality. W3. The value-learning targets may be suboptimal. The backward-generated trajectories are guaranteed to reach the identity, but they are not necessarily short. Random action sequences may contain redundant operations, making their return-to-go labels unreliable estimates of the shortest synthesis cost. It remains unclear why learning from these labels should lead to efficient circuits. A comparison with behavioral cloning or direct inverse-action prediction would help clarify the benefit of value learning. W4. The compilation overhead is substantial. According to Appendix F, F2 requires 11–24 seconds for 12-qubit tasks and 141–295 seconds for 144-qubit tasks, while conventional compilers generally finish in under 1.3 seconds. Despite running on an A100 GPU, F2 is several orders of magnitude slower. The paper does not sufficiently explain when the reduction in RZ gates would justify this additional compilation cost. W5. The RL objective is not well aligned with actual circuit cost. F2 assigns a reward of −1 per action, effectively minimizing the number of abstract synthesis steps rather than the final gate count or circuit depth. Since different actions can have different implementation costs, shorter action sequences do not necessarily produce more efficient circuits. The lack of CX gate counts also makes it difficult to assess the practical hardware benefits.

Questions for authors

Q1. Could the authors compare F2 with a deterministic free-fermionic compiler under the same partitioning and post-processing pipeline to isolate the contribution of RL? Q2. What fraction of the Hamiltonian is actually compiled by F2 in the larger benchmarks, and how do cross-window terms affect the results? Q3. Why should return-to-go labels from random, potentially redundant trajectories provide reliable guidance for short-circuit synthesis? How would behavioral cloning perform on the same data? Q4. How does the implementation handle numerical instability in the Cayley transform when (I+S) is nearly singular? Q5. Could the authors report full-circuit fidelity, compilation success rates, and CX gate counts across all benchmarks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.