- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces GenQAS, a reinforcement learning framework for quantum architecture search that combines a tensor-network-based warm start with prioritized generative replay. The key idea is to train a model of circuit transitions, rewards, and termination from collected experience, and use its predictions to augment Double DQN updates without requiring additional quantum circuit evaluations.
The authors motivate the approach through the increasing sparsity of useful circuit trajectories as the system size grows, provide a theoretical analysis of sample starvation, and evaluate GenQAS on molecular Hamiltonians, a transverse-field Ising model, and Clifford circuit synthesis.The experimental results are encouraging. The method achieves substantial improvements in success probability relative to uniform and prioritized replay, particularly at 12 qubits, where the reported improvement reaches approximately 7×. The transfer experiments under gate noise also demonstrate promising improvements in sample efficiency.
Critical strengths
-
Well-motivated application of generative replay. The paper identifies an important practical bottleneck in RL-based quantum architecture search. Using learned models to reduce reliance on expensive quantum evaluations is a natural and potentially valuable direction.
-
Strong empirical improvements. The improvements over passive replay are substantial, particularly on the larger molecular benchmarks. The comparison against both uniform and prioritized replay, using the same tensor-network initialization and RL backbone, helps isolate the contribution of the replay mechanism.
-
Breadth of evaluation. The experiments include several molecular systems, a 15-qubit spin model, noisy transfer, and a discrete Clifford synthesis task. The inclusion of circuit resource statistics, generator diagnostics, and wall-clock measurements is also appreciated.
Critical weaknesses
-
*The theoretical claims appear stronger than the provided justification. In particular, the proof of Theorem 3.5 seems to attribute part of the convergence improvement to the MPS initialization, despite the experimental baselines sharing this initialization. I also have concerns about some of the assumptions and concentration arguments in Appendix A. The claimed exponential sample-complexity improvement does not appear sufficiently established for the actual algorithm.
-
The mechanism underlying the improvement could be better isolated. The synthetic transitions are generated from previously observed state-action pairs. Consequently, it is unclear to what extent the improvement should be understood as expanding informative experience, as opposed to improving value estimation through model-based augmentation of existing samples. More direct measurements of the generated transitions' usefulness, or an ablation of the individual prediction heads, would help establish the mechanism.
-
Circuit resource comparisons require clarification. The reported circuit statistics appear to describe the learned circuit suffix, while the fixed MPS preparation circuit introduces additional gates and depth. Although this does not invalidate the comparisons between GenQAS and TensorRL-QAS, which share the same initialization, comparisons against other circuit architectures should account for the complete executable circuit.
Questions for authors
-
Can the authors clarify how Theorem 3.5 establishes an improvement specifically attributable to generative replay when both GenQAS and the passive-replay baselines use the same MPS warm start?
-
Since generated transitions retain real state-action seeds, how does generative replay increase the support of informative experience? Have the authors measured whether synthetic transitions provide information beyond additional updates using the corresponding real transitions?
2 comments
Official Comment By Authors
Dear Reviewer,
Thank you again for your review, and we fully understand that the discussion period comes on top of many other commitments.
We have posted a detailed response to each of your points, with the additional experiments you asked for. As the discussion period is coming to a close, we would be grateful to hear whether our response addresses your concerns. If any point still seems unresolved, please let us know, we are glad to clarify or provide further clarification before the discussion ends.
Best regards,
The Authors
Response to your comments and suggestions
W → Weakness; Q → Questions. All the changes mentioned below are highlighted in red in the revised manuscript.
(W1 and Q1) Attribution of convergence speedup under matched warm starts and unconditioned generation.
We thank the reviewer for this constructive critique and fully agree: our original Theorem 3.5 conflated the warm start with generative replay. In revised Section 3.3 and Appendix A.5, we downgrade it to Proposition 3.6 under matched warm starts at sTN, matching initial suboptimality gaps Δ0∈O(1) and iteration steps K. Speedup does not stem from initial value iteration; instead, as formalized in Lemma 3.5 (Appendix A.6), localized dimension deff∼2N+1 causes passive real transition density to decay as ρreal(N)∼exp(−c⋅2N). For an unconditioned 1-step model fψ(s,a) with generalization error ϵmodel, the sample complexity ratio satisfies: STensorRL(ϵc)SGenQAS(ϵc)≤1−Gr+Grκ(N)1,whereκ(N)≜ρreal(N)ρsyn(N)(1−O(ϵcϵmodel)) This proves that acceleration (κ>1) holds only when transition starvation dominates (N≥10), while model bias (ϵmodel≥ϵc) yields κ<1, explaining why Gr=0.8 harms small systems (0.28 vs.\ 0.60 at 6q). Finally, our new cold-start ablation confirms that generative replay independently reaches chemical accuracy from ∣0⟩⊗N across 6-10 qubit (78.6%, 69.0%, 35.0%), validating the global motivation when unanchored which is in Appendix D.9.
(W2 and Q2) Isolating the improvement mechanism: expanding experience versus value-estimation regularization.
To isolate whether GenQAS expands informative experience or regularizes value estimation, we synthesized findings from two targeted ablations: matched-Gr Prioritized Duplicate Replay and the cold-start ablation.
First, Prioritized Duplicate Replay (Gr=0.4 on 8q H2O) tests whether synthetic transitions act merely as repeated real updates. Duplicating real transitions completely fails to reach chemical accuracy (2.43×10−3 Ha) and bloats depth to 11 with 12–13 CNOTs, whereas GenQAS reaches 1.19×10−3 Ha (84.5% success) with 3 CNOTs (depth 4). Mechanistically, exact sample duplication creates rigid delta-peaks that induce Q-value overestimation and policy memorization on narrow paths, whereas the learned forward model fψ(s,a) smooths representations across continuous variational parameters and state projections, acting as a functional Bellman regularizer.
Second, evaluating cold-start generative replay from ∣0⟩⊗N across 6–10 qubits tests experience expansion without an inductive prior. While passive exploration collapses, generative replay alone reaches chemical accuracy on 6q (78.6%), 8q (69.0%), and 10q (35.0%), proving on-demand generation actively propagates rare transition signal along high-priority gradients to expand reachable support. Paired with noisy transfer experiments (Section 4.2, where generative buffers yield a 92.7% step reduction under noise vs. 30.1% for passive replay), this confirms GenQAS operates by both expanding reachable support and regularizing value estimation (Section 4.5, Appendix D.10).
(W3) Circuit resource comparisons accounting for the complete executable circuit.
We thank the reviewer for this constructive observation. We fully agree: while Table 1 reports the learned suffix (Vt) to isolate search efficiency between RL algorithms sharing the warm start, comparisons against external baselines like UCCSD and HEA must account for the full executable circuit (U=VtUTN).
Analogous to starting VQE from Hartree-Fock rather than ∣0⟩⊗N, GenQAS initializes from an approximate MPS state compiled into fixed prefix UTN. Appendix Table 3 reports its exact footprint (fixed depth 27; 15, 21, and 27 CNOTs for 6q, 8q, and 10q). Even accounting for the full executable circuit (UTN+Vt), GenQAS maintains decisive advantages over UCCSD: on 6q BeH2, it uses 20 CNOTs at depth 32 (13.6× fewer CNOTs, 14.8× shallower); on 8q H2O, it uses 24 CNOTs at depth 31 (54.6× fewer CNOTs, 69.3× shallower, where HEA fails chemical accuracy); and on 10q H2O, it uses 41 CNOTs at depth 47 (83.9× fewer CNOTs, 113.5× shallower). We retain Table 1 focused on the learned suffix for clean RL comparison, but expanded Section 4.3 and Appendix B.6 to discuss complete executable circuit statistics.