acceptodds
Withdrawn from ICLR 2027

Classical State Preparation for Variational Quantum Algorithms via Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Variational Quantum Algorithms (VQAs) potentially offer a pathway to practical quantum advantage, but their optimization is heavily hindered by barren plateaus and numerous local minima. While classically simulable Clifford circuits can warm-start VQAs to accelerate convergence, existing heuristic-based initialization methods struggle to scale within vast combinatorial search spaces. We address this bottleneck with CRiSP (a Clifford Reinforcement Learning agent for State Preparation), which casts Clifford prefix selection as a sequential decision-making problem. CRiSP utilizes Neural-Guided Monte Carlo Tree Search, driven by a Transformer-based policy trained via self-play, to insert learned Clifford gates before fixed parameterized rotations. This enables the construction of high-quality initial states entirely through polynomial-time classical stabilizer simulation without altering the underlying circuit architecture. By integrating a curriculum learning strategy that progressively expands the search horizon, the agent efficiently scales to deep circuits. Evaluated on QAOA benchmarks of up to qubits and parameters, CRiSP outperforms state-of-the-art Clifford initialization methods by a mean of (max ) in average energy accuracy and (max ) in best-achieved energy accuracy. Assessments on VQE tasks further demonstrate the framework's robustness and generalizability.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Quantum Computing, Variational Quantum Algorithms, Classical State Preparation, Reinforcement Learning, Monte Carlo Tree Search, Clifford Circuits, QAOA, Curriculum Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8dn6OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce CRiSP, a classical warm start framework for VQAs that casts Clifford initialization as a sequential decision problem. Under this process, a discrete Clifford gate is add to the front to each rotation of a fixed ansatz. With all continuous parameters set to zero, the circuit is Clifford, so its energy can be computed exactly with a stabilizer simulator. The prefixes are chosen by an AlphaZero-style neural guided MCTS agent with a Transformer policy value network, replay and best-game buffers, and an incremental horizon curriculum. The resulting Clifford state serves as the starting point for downstream continuous optimization, and the search itself needs no quantum evaluations. The authors frame the motivation around two problems of existing Clifford initializers (1) the Clifford search space grows exponentially with the number of circuit parameters, so Bayesian and genetic algorithm heuristics struggle on challenging landscapes. (2) VQA optimization from arbitrary starting points is unreliable because of local minima and barren plateaus, and every quantum evaluation is costly, so a good classical initialization is valuable. The method is evaluated on QAOA tasks with up to 22 qubits and 1370 parameters against SPIQ, and on a preliminary VQE study with 10-qubit Ising and XXZ chains against a GA variant of CAFQA.

Critical strengths
  1. The paper is quite well written and content is easy to follow. The motivation of the paper classically i.e simulable Clifford warm starts for VQAs, is quote straightforward. The appendix gives further hyperparameters and implementation details.

  2. The overview figure of the framework conveys the key idea quickly and makes the method accessible to readers.

  3. The evaluation is broad. It covers 14 QAOA instances across 3 problem families up to 22 qubits. In addition a VQE study on Ising and XXZ models is provided.

Critical weaknesses
  1. The work reads more of an incremental contribution. Clifford-based classical initialization is already established through CAFQA, Clapton, SPIQ which the author refer in the paper. I believe that the proposed method combines standard components such as AlphaZero-style neural guided (NG) MCTS, a Transformer policy-value network, replay and best-game buffers as in AlphaTensor, and a horizon curriculum. The contribution is effectively just replacing a genetic algorithm search with NG-MCTS inside a known pipeline, and the closest RL-based prior work such as MCTS-QAOA is distinguished only briefly.

  2. Although the tasks are diverse which I also meantioned in the strengths, QAOA is compared only against one method SPIQ which is a genetic algorithm and VQE only against a GA variant of CAFQA. There is no comparison against other Clifford initializers e.g., Clapton, other RL approaches such as [Kremer, David, et al. "Practical and efficient quantum circuit synthesis and transpiling with reinforcement learning." arXiv preprint arXiv:2405.13196 (2024).], or nonClifford state-of-the-art warm starts (kindly see the next weakness), so I think the claim of state-of-the-art performance rests on one baseline per setting.

  3. Following the above point, the VQE evaluation is limited in scope and in strength of evidence. The study covers only two 10-qubit 1D spin models and the method provides a geometric-mean gain of about 1.1 in average accuracy and 1.11 in best-achieved accuracy, which is a small improvement. Moreover, tensor-network-based initialization e.g., Rudolph et al., which the appendix cites, and warm-started RL for circuit discovery e.g., TensorRL-QAS, NeurIPS 2025 are not benchmarked. This matters because 1D chains at this size are precisely the regime where tensor-network methods are near-exact, so it remains unclear whether a Clifford warm start offers any advantage over cheap classical alternatives. To assess the method's real capability, I encourage the authors to include benchmarks where tensor networks are not trivially sufficient, such as molecular Hamiltonians or 2D models.

  4. All results come from noiseless stabilizer simulation. The paper does not test whether Clifford-based initializations remain better under hardware noise, although noise-aware Clifford methods exist and downstream optimization is run on noisy devices in practice.

  5. The paper states that CRiSP "exhibits strong scalability," yet it trains a Transformer policy value network with MCTS separately for each instance, taking 10-80 hours on an A100 GPU with 32 CPU cores. The authors acknowledge that training cost grows with system size and that the search space remains exponential, and accuracy on MaxCut drops with size. The time-matched comparison is only against a GA baseline, so it is unclear whether simpler alternatives, such as spending the same compute on additional restarts of the downstream optimizer, would give a similar benefit. A full accounting of classical training cost per instance versus the quantum evaluations saved, and a scaling study against parameter count, are needed.

  6. Three seeds per task is thin for a stochastic search method, and the paper reports wider variance on sparse tasks, where CRiSP sometimes underperforms the baseline on average. Significance tests and more seeds would strengthen the claims.

Questions for authors
  1. Beyond just replacing the genetic algorithm with NG-MCTS and using a Transformer policy value network in the Clifford space, what is the concrete algorithmic novelty relative to MCTS-QAOA and other warm-started RL approaches?
  2. How does CRiSP compare against Clapton and other RL approaches such e.g., Kremer et al. (see weakness 2) under equal wall clock time and equal circuit evaluation budgets?
  3. (a) Can the authors report CRiSP against MPS/DMRG-based initialization (Rudolph et.al) and TensorRL-QAS on the same VQE instances? (b) Can you comment how CRiSP would perform where tensor networks are not near exact such chemical Hamiltonian or 2D models?
  4. Can the authors evaluate CRiSP under noise, on hardware if available, or otherwise in simulation with (i) depolarizing noise, (ii) dephasing noise, and (iii) finite-shot sampling and report the initial energy and the final energy after downstream optimization on the same QAOA and VQE instances?
  5. What is the total classical training cost which includes GPU hours + network evaluations per instance as the number of parameters grows, and what does a simple baseline, such as extra downstream restarts, achieve at the same cost?
  6. Do the improvements remain statistically significant with more seeds, especially on the sparse tasks where CRiSP sometimes underperforms on average?
Reviewer_3CNpOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CRiSP essentially a fast start for approaching good VQA ansatze by picking a good Clifford set as the starting point for variational quantum circuits. With every rotation angle set to a Clifford value the circuit can be simulated cheaply with Stim. The CRiSP's agent picks one of 24 single-qubit Cliffords at each position and the episode reward is −<H>-<H> of the resulting stabilizer state. Search uses an AlphaZero-style MCTS with a Transformer policy and value network trained by self-play. Training uses a quite heuristic curriculum whereby the agent starts on short circuits and is given longer ones as it improves. Experiments use ma-QAOA (with p=1p=1) on MaxCut, Knapsack and two medical PCBO tasks, up to 22 qubits and 1,370 parameters, with the reconfigured SPIQ as the main baseline. A small VQE study on 10-qubit TFIM and XXZ models compares against a GA version of CAFQA. The main results reported are 83.7% mean accuracy against 40.5% for SPIQ, a 3.19× geometric-mean gain, and 1.1× over CAFQA on VQE.

Critical strengths

The framing of the authors as a sequential decision problem is innovative,. clean, and the implementation looks careful. The ablations are the right ones and I am happy with thopse: unguided MCTS, random search, the curriculum (see above) and the best-game buffer. I was glad to see a time matched SPIQ baseline. On the benchmarks chosen, CRiSP beats SPIQ by a large and consistent margin, especially on the larger Knapsack and MedicalDense instances. In addition the paper is easy to read and follow.

Critical weaknesses

My main concern is that the QAOA results are really results on classical optimisation and they need benchmarked accordingly. I.e., MaxCut, Knapsack and PCBO cost Hamiltonians are all diagonal in ZZ. The best a Clifford circuit can do is reach the optimal bitstring, which a layer of single-qubit Cliffords can already prepare ok. Accuracy of Ebest/Eopt=100%E_{\mathrm{best}}/E_{\mathrm{opt}} = 100\% just means the instance has been solved, ie global optimum attained and the authors already know EoptE_{\mathrm{opt}} for every instance. In effect, the search runs over at most 2222^{22} bitstrings, which is small enough to enumerate via brute force (not done though). Really, the important comparisons to demonstrated alleged superiority are missing. There is no exhaustive search, nor Gurobi/MOSEK/CPLEX/HiGHS (they are cited in the appendix but never run), and no simulated annealing or greedy search on bitstrings. Against these, 10-80 A100-hours per instance (App. A) looks very expensive. The downstream QAOA warm start of Fig. 4b begins from what is essentially the answer, and it plots only the best run per method.

And then, where the problem really is quantum, the gain is quite small. On non-diagonal TFIM and XXZ at n=10n=10 CRiSP gets 77.93% against 73.97% for CAFQA. That covers two models and three seeds, with a tie on Ising J=1.0J=1.0. Several entries show a std of exactly 0.00 which is problematic and needs explaining.

In addition, the statistics as performed flatter the method. Ratio improvements such as the 45× maximum come from near-zero baselines, for example SPIQ at 2.08% on MedicalDense-20. MedicalSparse has std up to 45 points over three seeds, and the text argues that the best seed is what counts. Absolute differences with confidence intervals over more seeds would be more convincing.

Questions for authors
  1. How does CRiSP compare with brute force, Gurobi, or simulated annealing on bitstrings for the QAOA instances, in both solution quality and wall-clock time? Or even with HiGHS etc?

  2. Since the optimal stabilizer state for a diagonal HCH_C is a basis state, what does a QAOA warm start that already reaches EoptE_{\mathrm{opt}} add in practice?

  3. Can you please show results on non-diagonal targets at larger scale, such as molecular Hamiltonians or TFIM/XXZ with n≥16n \ge 16, against CAFQA and Clapton?

  4. Does a policy trained on one instance transfer to new instances from the same family?

Reviewer_gKmDOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Variational quantum optimization algorithms have the capability to obtain solution to many classically hard problems. However, the VQA algorithms suffer from plateaus and local minima. Every evaluation of the objective function requires either a NISQ hardware or a classical simulator. The simulator is often computationally expensive for example in the case of Clifford circuits. Good initialization of these algorithms are important for faster convergence. Several algorithms have been presented for determining the initialization states. In this work, authors propose use of a reinforcement learning based discrete optimization framework for searching the optimal initialization. The discrete optimization problem is that of a neural guided Monte Carlo tree search. Replay is used by the reinforcement learning algorithms. A transformer is used as the value function.

Experimental results are provided to demonstrate the effectiveness of the initialization scheme for the QAQA benchmark optimization tasks. Results show that the proposed method indeed generates some good initialization in terms of the final solution of the optimization problem to which the VQA converges.

Critical strengths

The problem of initialization of VQA algorithm is an important one in the NISQ horizon while using simulable Clifford circuit based approaches. This is due to high cost of every iteration requiring a simulator in the presence of noise. Although there are existing approaches of solving this problem, a new reinforcement learning based approach is proposed in the current manuscript. The approach uses an elegant Monte Carlo tree search like discrete optimization formulation of this initialization task and then uses a replay based RL framework. A transformer is used as the value function/policy. The use of RL in this setting is efficient. Results show benefits over existing approaches.

Critical weaknesses

The problem of initializing VQA algorithms is well studied in literature. The simulable Clifford gate based setup has particularly been extensively explored. Seceral approaches based on optimization in the classical setup has been used for this purpose. Authors present a reinforcement learning based approach instead. The rationale for this is not clear in the context of variational quantum algorithms. The paper lacks novelty in this aspect. There are several sophistication introduced in the RL algorithm used, but again their justifications are not well discussed. In current forms it just appears a direct addition of RL based search for the VQA initialization problem.

Experimental results are provided for the QAQA benchmark, but the superiority of the RL approach is not clear. The goodness of the initial states in terms of the final converged solution of the optimization algorithm is presented. But the time complexity of the RL approach or the convergence of the RL is not clear from experimental results. Many more experimental results are required including comparison with other RL setups are required to establish the effectiveness of the proposed CRiSP method.

Questions for authors
  1. Why is the particular RL approach used? Why not other RL formulations?
  2. Why is RL the preferred method? What are its advantages over traditional search based approaches that are commonly used in VQA structure optimization?
  3. How does the proposed method compare with state of art initialization schemes that use alternate circuit optimization techniques?
  4. More comparison in terms of the convergence time of the RL approach vis a vis the traditional and existing approaches should be studied.
  5. In general, many more ablation studies should be presented in the experimental results section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.