acceptodds
Under review as a conference paper at ICLR 2027

Generative Replay Mitigates Sample Starvation in Quantum Architecture Search

Abstract

Reinforcement learning (RL) can automate quantum architecture search (QAS), but its scalability breaks down when useful circuit trajectories become rare in a rapidly expanding search space. We show that the probability of encountering an -accurate low energy circuit through passive exploration decreases super exponentially with qubit count, making uniform and prioritized experience replay fundamentally insufficient for large-scale QAS. We introduce GenQAS, a tensor network-guided RL framework that combines a fixed matrix product state warm start with prioritized generative replay. In this framework, a learned local transition model generates synthetic circuit transitions on demand from real state-action interactions and mixes them with real experience during Double Deep Q-Network updates. Across chemical Hamiltonian benchmarks from 6 to 12 qubits, GenQAS improves fixed-budget success probability and identifies more compact circuits at competitive energy error, improving final success probability by up to 7.0 over passive replay at 12 qubits. On a 15-qubit transverse field Ising model, GenQAS reaches 21% success probability, a 75% relative gain over uniform TensorRL-QAS (12%) and a 40% relative gain over prioritized TensorRL-QAS (15%). To test transfer, we move a replay buffer trained in a noiseless 6-qubit environment into a noisy version of the same task, comparing a transferred uniform buffer against a transferred GenQAS generative buffer. The uniform transfer reduces steps to chemical accuracy by only 30.1%, whereas the GenQAS generative replay transfer reduces steps by 92.7%, a substantially larger improvement under identical noise conditions. These results show that generative replay mitigates sample starvation in QAS and, critically, that the learned generative buffer carries far more transferable structure than passive replay, supporting more resource-efficient and robust circuit discovery. The code is available at https://anonymous.4open.science/r/GenQAS/README.md.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
We introduce GenQAS: an RL-framework combining tensor network warm start with prioritized generative replay to overcome sample starvation in reinforcement learning-based quantum architecture search, achieving up to 7.0$\times$ higher success and 92.7\% fewer steps in noisy transfer.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sj4hOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces GenQAS, a reinforcement learning framework for quantum architecture search that combines a tensor-network-based warm start with prioritized generative replay. The key idea is to train a model of circuit transitions, rewards, and termination from collected experience, and use its predictions to augment Double DQN updates without requiring additional quantum circuit evaluations.

The authors motivate the approach through the increasing sparsity of useful circuit trajectories as the system size grows, provide a theoretical analysis of sample starvation, and evaluate GenQAS on molecular Hamiltonians, a transverse-field Ising model, and Clifford circuit synthesis.The experimental results are encouraging. The method achieves substantial improvements in success probability relative to uniform and prioritized replay, particularly at 12 qubits, where the reported improvement reaches approximately 7×. The transfer experiments under gate noise also demonstrate promising improvements in sample efficiency.

Critical strengths
  1. Well-motivated application of generative replay. The paper identifies an important practical bottleneck in RL-based quantum architecture search. Using learned models to reduce reliance on expensive quantum evaluations is a natural and potentially valuable direction.

  2. Strong empirical improvements. The improvements over passive replay are substantial, particularly on the larger molecular benchmarks. The comparison against both uniform and prioritized replay, using the same tensor-network initialization and RL backbone, helps isolate the contribution of the replay mechanism.

  3. Breadth of evaluation. The experiments include several molecular systems, a 15-qubit spin model, noisy transfer, and a discrete Clifford synthesis task. The inclusion of circuit resource statistics, generator diagnostics, and wall-clock measurements is also appreciated.

Critical weaknesses
  1. *The theoretical claims appear stronger than the provided justification. In particular, the proof of Theorem 3.5 seems to attribute part of the convergence improvement to the MPS initialization, despite the experimental baselines sharing this initialization. I also have concerns about some of the assumptions and concentration arguments in Appendix A. The claimed exponential sample-complexity improvement does not appear sufficiently established for the actual algorithm.

  2. The mechanism underlying the improvement could be better isolated. The synthetic transitions are generated from previously observed state-action pairs. Consequently, it is unclear to what extent the improvement should be understood as expanding informative experience, as opposed to improving value estimation through model-based augmentation of existing samples. More direct measurements of the generated transitions' usefulness, or an ablation of the individual prediction heads, would help establish the mechanism.

  3. Circuit resource comparisons require clarification. The reported circuit statistics appear to describe the learned circuit suffix, while the fixed MPS preparation circuit introduces additional gates and depth. Although this does not invalidate the comparisons between GenQAS and TensorRL-QAS, which share the same initialization, comparisons against other circuit architectures should account for the complete executable circuit.

Questions for authors
  1. Can the authors clarify how Theorem 3.5 establishes an improvement specifically attributable to generative replay when both GenQAS and the passive-replay baselines use the same MPS warm start?

  2. Since generated transitions retain real state-action seeds, how does generative replay increase the support of informative experience? Have the authors measured whether synthetic transitions provide information beyond additional updates using the corresponding real transitions?

2 comments
AuthorsOpenReview ↗

Response to your comments and suggestions

W →\rightarrow Weakness; Q →\rightarrow Questions. All the changes mentioned below are highlighted in red in the revised manuscript.

(W1 and Q1) Attribution of convergence speedup under matched warm starts and unconditioned generation.

We thank the reviewer for this constructive critique and fully agree: our original Theorem 3.5 conflated the warm start with generative replay. In revised Section 3.3 and Appendix A.5, we downgrade it to Proposition 3.6 under matched warm starts at sTNs_{\mathrm{TN}}, matching initial suboptimality gaps Δ0∈O(1)\Delta_0 \in \mathcal{O}(1) and iteration steps KK. Speedup does not stem from initial value iteration; instead, as formalized in Lemma 3.5 (Appendix A.6), localized dimension deff∼2N+1d_{\mathrm{eff}} \sim 2^{N+1} causes passive real transition density to decay as ρreal(N)∼exp⁡(−c⋅2N)\rho_{\mathrm{real}}(N) \sim \exp(-c \cdot 2^N). For an unconditioned 1-step model fψ(s,a)f_\psi(s,a) with generalization error ϵmodel\epsilon_{\mathrm{model}}, the sample complexity ratio satisfies: SGenQAS(ϵc)STensorRL(ϵc)≤11−Gr+Grκ(N),whereκ(N)≜ρsyn(N)ρreal(N)(1−O(ϵmodelϵc))\frac{\mathcal{S}_{\mathrm{GenQAS}}(\epsilon_c)}{\mathcal{S}_{\mathrm{TensorRL}}(\epsilon_c)} \le \frac{1}{1 - G_r + G_r \kappa(N)}, \quad \text{where} \quad \kappa(N) \triangleq \frac{\rho_{\mathrm{syn}}(N)}{\rho_{\mathrm{real}}(N)}\left(1 - \mathcal{O}\left(\frac{\epsilon_{\mathrm{model}}}{\epsilon_c}\right)\right) This proves that acceleration (κ>1\kappa > 1) holds only when transition starvation dominates (N≥10N \ge 10), while model bias (ϵmodel≥ϵc\epsilon_{\mathrm{model}} \ge \epsilon_c) yields κ<1\kappa < 1, explaining why Gr=0.8G_r=0.8 harms small systems (0.280.28 vs.\ 0.600.60 at 6q). Finally, our new cold-start ablation confirms that generative replay independently reaches chemical accuracy from ∣0⟩⊗N|0\rangle^{\otimes N} across 6-10 qubit (78.6%78.6\%, 69.0%69.0\%, 35.0%35.0\%), validating the global motivation when unanchored which is in Appendix D.9.

(W2 and Q2) Isolating the improvement mechanism: expanding experience versus value-estimation regularization.

To isolate whether GenQAS expands informative experience or regularizes value estimation, we synthesized findings from two targeted ablations: matched-GrG_r Prioritized Duplicate Replay and the cold-start ablation.

First, Prioritized Duplicate Replay (Gr=0.4G_r = 0.4 on 8q H2O\mathrm{H}_2\mathrm{O}) tests whether synthetic transitions act merely as repeated real updates. Duplicating real transitions completely fails to reach chemical accuracy (2.43×10−3 Ha2.43\times 10^{-3}\text{ Ha}) and bloats depth to 11 with 12–13 CNOTs, whereas GenQAS reaches 1.19×10−3 Ha1.19\times 10^{-3}\text{ Ha} (84.5%84.5\% success) with 3 CNOTs (depth 4). Mechanistically, exact sample duplication creates rigid delta-peaks that induce Q-value overestimation and policy memorization on narrow paths, whereas the learned forward model fψ(s,a)f_\psi(s,a) smooths representations across continuous variational parameters and state projections, acting as a functional Bellman regularizer.

Second, evaluating cold-start generative replay from ∣0⟩⊗N|0\rangle^{\otimes N} across 6–10 qubits tests experience expansion without an inductive prior. While passive exploration collapses, generative replay alone reaches chemical accuracy on 6q (78.6%78.6\%), 8q (69.0%69.0\%), and 10q (35.0%35.0\%), proving on-demand generation actively propagates rare transition signal along high-priority gradients to expand reachable support. Paired with noisy transfer experiments (Section 4.2, where generative buffers yield a 92.7%92.7\% step reduction under noise vs. 30.1%30.1\% for passive replay), this confirms GenQAS operates by both expanding reachable support and regularizing value estimation (Section 4.5, Appendix D.10).

(W3) Circuit resource comparisons accounting for the complete executable circuit.

We thank the reviewer for this constructive observation. We fully agree: while Table 1 reports the learned suffix (VtV_t) to isolate search efficiency between RL algorithms sharing the warm start, comparisons against external baselines like UCCSD and HEA must account for the full executable circuit (U=VtUTNU = V_t U_{\mathrm{TN}}).

Analogous to starting VQE from Hartree-Fock rather than ∣0⟩⊗N|0\rangle^{\otimes N}, GenQAS initializes from an approximate MPS state compiled into fixed prefix UTNU_{\mathrm{TN}}. Appendix Table 3 reports its exact footprint (fixed depth 27; 15, 21, and 27 CNOTs for 6q, 8q, and 10q). Even accounting for the full executable circuit (UTN+VtU_{\mathrm{TN}} + V_t), GenQAS maintains decisive advantages over UCCSD: on 6q BeH2\mathrm{BeH}_2, it uses 2020 CNOTs at depth 3232 (13.6×13.6\times fewer CNOTs, 14.8×14.8\times shallower); on 8q H2O\mathrm{H}_2\mathrm{O}, it uses 2424 CNOTs at depth 3131 (54.6×54.6\times fewer CNOTs, 69.3×69.3\times shallower, where HEA fails chemical accuracy); and on 10q H2O\mathrm{H}_2\mathrm{O}, it uses 4141 CNOTs at depth 4747 (83.9×83.9\times fewer CNOTs, 113.5×113.5\times shallower). We retain Table 1 focused on the learned suffix for clean RL comparison, but expanded Section 4.3 and Appendix B.6 to discuss complete executable circuit statistics.

AuthorsOpenReview ↗

Official Comment By Authors

Dear Reviewer,

Thank you again for your review, and we fully understand that the discussion period comes on top of many other commitments.

We have posted a detailed response to each of your points, with the additional experiments you asked for. As the discussion period is coming to a close, we would be grateful to hear whether our response addresses your concerns. If any point still seems unresolved, please let us know, we are glad to clarify or provide further clarification before the discussion ends.

Best regards,

The Authors

Reviewer_wWxSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper argues that RL-based quantum architecture search suffers from the phenomenon of sample starvation ie as the qubit count grows, transitions that lower the energy become rarer and rarer and uniform or prioritised replay can only reuse what was already collected earlier on. The proposed GenQAS builds on TensorRL-QAS (which is a fixed MPS warm-start prefix followed by a DDQN RL agent that appends gates). It adds an MLP model trained on real transitions that generates synthetic transitions from real seeds in each minibatch during traiing. Then, a tunable ratio controls how much of each training batch is synthetic following the prioritised generative replay approach. Considering the theory the authors argue that low- nergy states make up a vanishingly small fraction of all possible states as the system grows (Thms 3.1–3.2) which is not as surprising, really well known. Eventually they frame the MPS warm start as a kind of guided generation, and a final result claims the method converges faster.

The suggested method is then tested on several molecules and a spin model at up to 15 qubits. The main result claimed is a much higher success rate than standard replay on the largest molecule, with smaller gains elsewhere and faster adaptation to a noisy setting.

Critical strengths

Indeed the problem the authors want to address is real: RL-QAS is expensive per transition and therefore getting more out of each real interaction is worth studying in detail. The comparison with TensorRL-QAS is well motivated and reasonably controlled with the same warm start and backbone and action space, so the replay mechanism is the main thing that changes. Also the paper is open about the related costs eg in table 2 it reports wall-clock time per episode, and the authors say plainly that generative replay is not a universal upgrade and that G_r has to be properly tuned. Hyperparameters are listed and an anonymised code link is provided. The gains at 6-10 qubits are consistent across sizes.

Critical weaknesses

The synthetic data do not really add anything new and do not help the author's case. This data is generated from real examples already in the batch, and the next circuit is fully determined anyway, so the model mostly re-predicts rewards the agent has already seen! The demonstrated small gains could just as easily come from replaying good examples more often for example. Simple comparisons that would settle this are missing eg duplicating real samples at the same ratio or replaying more often with standard prioritised replay. And this would be a good sanity test.

In addition, the theory does not match the experiments where the speedup result depends on only one method getting the warm start, but in the experiments both get it. Also the proofs assume a smarter, targeted generator than the one actually used in the experiments and I would read the theory section as motivation rather than as guarantees.

Finallt, the 12-qubit narrative is quite overstated. The issue is that the success bar at 12 qubits is about 16 times looser than for the smaller molecules and this is loose enough that even the simple baselines pass it, and no method gets close to chemical accuracy. The reporting is also optimistic too and several numbers also disagree between the intro, the tables and the appendix and the authors should fix for consistency.

Overall, this is a known idea (ie generative replay) applied to an existing QAS method, and the evidence does not at this instance show why or when it helps.

Questions for authors

(1) Since the next circuit state is deterministic given (s,a)(s,a) and seeds come from the current real batch, what information do synthetic transitions carry beyond a repredicted reward for an already seen pair? Could you maybe try run the ablations at matched GrG_r, i.e., duplicated real samples, PER with a higher replay ratio and Gaussian-perturbed rewards?

(2) Can Thrm 3.5 be restated for the comparison you actually run, with both arms warm-started and an unconditioned generator? What does κ\kappa look like then?

(3) Do the 12-qubit alleged gains hold at the 1.6×10−31.6\times10^{-3} threshold used for the smaller systems? Please maybe try to include mean ±\pm CI over seeds with GrG_r chosen on a validation criterion.

6 comments
AuthorsOpenReview ↗

A brief response to reviewer comments and suggestions

W →\rightarrow Weakness; Q →\rightarrow Questions, All the changes are highlighted in red in the revised manuscript.

(W1 and Q1) Role of deterministic transitions and duplicated samples.

We evaluated Prioritized Duplicate Replay (Duplicate PER) at matched Gr=0.4G_r = 0.4 on 8q H2O\mathrm{H}_2\mathrm{O}, duplicating real transitions instead of querying fψf_\psi:

Method / SettingReplay TypeMin Err (Ha)Chemical accuracy (<1.6 mHa)CXDepthTime/Ep
TensorRL-QAS (PER)Passive (Gr=0.0G_r=0.0)1.26×10−31.26 \times 10^{-3}Reached (57.3%)14111.17s
Duplicate PER (Seed 1)Duplication (Gr=0.4G_r=0.4)2.37×10−32.37 \times 10^{-3}Failed12102.57s
Duplicate PER (Seed 42)Duplication (Gr=0.4G_r=0.4)2.49×10−32.49 \times 10^{-3}Failed13112.96s
GenQAS (Gr=0.4G_r=0.4)Generative Replay1.19×10−31.19 \times 10^{-3}Reached (84.5%)342.41s

Duplicating real samples fails to reach chemical accuracy (2.43 mHa2.43\text{ mHa}) and bloats depth to 10–11 with 12–13 CNOTs, proving gains do not stem from oversampling observed data. Exact duplication creates delta-peaks that cause Q-value overestimation, whereas fψf_\psi smooths representations across continuous parameters, providing functional Bellman regularization (Section 4.3, Appendix D.10).

(W2 and Q2) Theory under matched warm starts and unconditioned generation.

We agree that Theorem 3.5 was formulated under idealized conditions (energy conditioning, cold-start comparison) as a conceptual motivation for escaping super-exponential starvation. In revised Section 3.3 and Appendix A.5, we downgrade it to Proposition 3.6 under matched warm starts at sTNs_{\mathrm{TN}}, where local dimension deff∼2N+1d_{\mathrm{eff}} \sim 2^{N+1} causes passive real transition density to decay as ρreal(N)∼e−c2N\rho_{\mathrm{real}}(N) \sim e^{-c 2^N}. For unconditioned fψf_\psi, the complexity ratio satisfies SGenQAS≤[1−Gr+Grκ(N)]−1STensorRL\mathcal{S}_{\mathrm{GenQAS}} \le [1 - G_r + G_r \kappa(N)]^{-1} \mathcal{S}_{\mathrm{TensorRL}} with κ(N)≈ρsynρreal(1−O(ϵmodelϵc))\kappa(N) \approx \frac{\rho_{\mathrm{syn}}}{\rho_{\mathrm{real}}}(1 - \mathcal{O}(\frac{\epsilon_{\mathrm{model}}}{\epsilon_c})). When starvation dominates (N≥10N \ge 10), κ>1\kappa > 1, making Gr=0.8G_r = 0.8 essential; when model error dominates (N=6N=6), κ<1\kappa < 1, explaining degradation under high GrG_r (0.280.28 vs. 0.600.60). Cold-start runs confirm generative replay reaches chemical accuracy from ∣0⟩⊗N|0\rangle^{\otimes N} across 6-10-qubit (78.6%, 69.0%, 35.0%), validating global motivation.

(W3 and Q3) 12-qubit narrative, threshold basis, and multi-seed statistics.

We agree our 12q claims were overstated: neither GenQAS nor baselines reach chemical accuracy (1.6×10−3 Ha1.6\times 10^{-3}\text{ Ha}) at 12q, which requires fermionic excitation [Nat comm 10.1 (2019): 3007.] or Majorana operators [arXiv:2503.18939 (2025).]. Our 2.5×10−2 Ha2.5\times 10^{-2}\text{ Ha} threshold was calibrated to the best classical simulated annealing (SA) result under the same gate set, where GenQAS demonstrates resilience to starvation (peak 79% vs. 11% for PER, 7.2×7.2\times gain). Across 5 seeds with Gr∗G_r^* chosen via validation TD loss:

SystemThreshold (Ha)BasisUniformPERGenQAS (Gr=0.4G_r=0.4)GenQAS (Gr=0.8G_r=0.8)Validation Gr∗G_r^*
6-BeH2BeH_21.6×10−31.6\times 10^{-3}Chemical Accuracy58.84 ±\pm 0.74%60.40 ±\pm 0.47%88.63 ±\pm 0.45%27.00 ±\pm 0.73%Gr∗=0.4G_r^* = 0.4
8-H2O{H}_2O1.6×10−31.6\times 10^{-3}Chemical Accuracy55.80 ±\pm 0.67%57.30 ±\pm 0.26%84.54 ±\pm 0.14%35.94 ±\pm 0.34%Gr∗=0.4G_r^* = 0.4
10-H2OH_2O1.6×10−31.6\times 10^{-3}Chemical Accuracy51.08 ±\pm 0.44%55.08 ±\pm 0.24%84.37 ±\pm 0.29%21.38 ±\pm 0.16%Gr∗=0.4G_r^* = 0.4
10-H2OH_2O2.5×10−22.5\times 10^{-2}Best SA (HEA)8.35 ±\pm 0.14%11.94 ±\pm 0.15%36.72 ±\pm 0.24%76.78 ±\pm 0.24%Gr∗=0.8G_r^* = 0.8

Thresholds and multi-seed stats are added to Figure 1 caption and Section 4.1.

(W4) Algorithmic framing, and evidence for why and when GenQAS helps.

Our contribution diagnoses sample starvation in RL-QAS and establishes why and when on-demand replay succeeds: Why: Duplicate PER shows oversampling observed data fails (2.432.43 mHa, depth 11), while fψf_\psi regularizes representations (1.191.19 mHa, depth 4). In noisy transfer (Section 4.2), generative buffers yield a 92.7%92.7\% step reduction vs. 30.1%30.1\% for passive replay. When: At small NN with abundant data, high GrG_r causes model bias (0.280.28 at 6q); under severe starvation at 12q (11% passive success), high GrG_r is vital (79%, 7.2×7.2\times gain). Cold start proves generative replay functions without warm starts (35%-78.6%), while MPS warm starts prevent scalability collapse (84% vs. 35% at 10-qubit) and cut runtime by 79.2%-94.9%.

Thanks for tthe comments and the new experiments. Ok reallt (i) the 5-seed statistics, (ii) the clearer 12-qubit threshold and the (iii) matched warm start version of the theory are improvements to some extent however my main concerns remain so I decide to keep my score.

First, the mechanism is still quite unclear. Of the three asks I asked for, only sample duplication was run as far as I can tell, on one system with 2 seeds. It seems to do worse than plain PER so it is a weak result. The new explanation, that the model smooths the TD updates is what the noisy-reward control would test. If that is the mechanism, sample starvation does not explain the gains.

Then the key step of Prop 3.6 is assumed but not proven. Synthetic transitions come from real (s, a) pairs already in the buffer, so it is unclear how they can be informative more often than the real data.

Finallt, I think that some numbers need checking. In Table 9, the 8- and 10-qubit H2O errors are identical at all three shot budgets, and shot noise gives 100% success against 84-89% without it. The confidence intervals in Table 4 look too tight for 5 RL seeds. Also your abstract still says 7.2x at 12 qubits, which uses the peak run and a threshold 16x looser than chemical accuracy. Fianally, the cold start ablation also lacks a passive cold start baseline as far as I can tell.

AuthorsOpenReview ↗

Response to reviewer part I

We sincerely thank you for acknowledging the improvements.In the following we provide our best to address your concerns.

First, the mechanism is still quite unclear.

Response: We appreciate the your important perspective on isolating the mechanism. Rather than a weak result, the phenomena that Prioritized Duplicate Replay performs significantly worse than plain PER is what rules out the primary alternative hypothesis: The gains raises from simply replaying observed high priority samples more frequently. When good transitions are exponentially sparse due to sample starvation. Hence repeatedly replaying identical observed points creates rigid delta-peaks in the Bellman loss [Lyle, Clare, Mark Rowland, and Will Dabney. "Understanding and preventing capacity loss in reinforcement learning." arXiv preprint arXiv:2204.09560 (2022)], causing severe Q-value overestimation and trapping the policy in redundant gate loops.

More importantly, smoothing of representation and sample starvation mitigation are not conflicting mechanisms and we believe that they are complementary. Sample starvation creates an extreme scarcity of informative transitions in the buffer. The learned model fψ(s,a)f_\psi(s,a) mitigates this scarcity by generalizing across continuous variational parameters and state projections, generating locally consistent counterfactual transitions around sparse high-value seeds without requiring costly new VQE evaluations. Adding isotropic Gaussian noise to rewards would simply perturb scalar targets on memorized transitions without modeling the coupled state-action dynamics or continuous geometry of the ansatz manifold. Together with our noisy buffer transfer experiment, where generative replay achieves a 92.7%92.7\% step reduction versus 30.1%30.1\% for passive buffers, this confirms that fψf_\psi extracts transferable functional circuit dynamics rather than acting as a superficial reward regularizer.

Then the key step of Prop 3.6 is assumed but not proven.

Response: We thank the reviewer for this incisive theoretical point. We completely agree: uniform sampling of buffer seeds could never yield ρsyn>ρreal\rho_{\mathrm{syn}} > \rho_{\mathrm{real}}. In revised Section 3.4 and Appendix A.5, we clarify that ρsyn>ρreal\rho_{\mathrm{syn}} > \rho_{\mathrm{real}} arises specifically because seeds are not drawn uniformly across the exploration history, but via prioritized sampling proportional to TD-error, concentrating generation around the rare high-priority frontier while passive memory remains diluted by ϵ\epsilon-greedy random walks. Furthermore, because the state space encodes continuous variational angles and energy projections, fψ(s,a)f_\psi(s,a) smooths representations across parameter manifolds, interpolating valid Bellman targets rather than duplicating discrete points.

We also agree that Proposition 3.6 serves as a conditional analytical model rather than an unconstrained guarantee. We have updated the asymptotic κN\kappa^N claim in Appendix A.5, explicitly stating that acceleration requires bounded local model error (ϵmodel<ϵc\epsilon_{\mathrm{model}} < \epsilon_c). When starvation dominates, κ(N)>1\kappa(N) > 1 yields speedup; when model error dominates or real data is abundant, κ(N)<1\kappa(N) < 1, rigorously explaining why high GrG_r harms small systems.

Reviewer_cEHfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

As qubit increases, it is required for super-exponential scarcity of low-energy circuit trajectories using RL based QAS (quantum architecture search). To address this challenge, GenQAS proposed to combine a fixed Matrix Product State (MPS) warm-start prefix from TensorRL-QAS with Prioritized Generative Replay (PGR). An MLP dynamics model generates 1-step synthetic transitions on demand from real seeds to mix into Double DQN updates without persisting in the replay buffer. Experiments cover 6–12 qubit molecules, a 15-qubit TFIM, and 4-qubit Clifford synthesis.

Critical strengths
  • GenQAS generates 1-step synthetic transitions on demand from real seeds without storing them in the persistent buffer prevents error compounding and policy collapse.

  • The proposed combined approach discovers circuits with substantially reduced CNOT counts on 8- and 12-qubit systems (e.g., 2–3 CNOTs vs. 14–18 for TensorRL-QAS) at comparable energy error.

  • GenQAS demonstrated a 92.7% reduction in search steps when transferring a noiseless replay buffer into a depolarizing noise environment.

Critical weaknesses
  • The ablation study is largely restricted to comparing GenQAS against its parent architecture, TensorRL-QAS (uniform and PER). It lacks systematic ablations against a broader range of baselines—such as applying generative replay directly to standard model-free QAS (without the MPS warm start), or comparing against alternative buffer management strategies.

  • The framework is a combination of three existing components: the MPS warm start from TensorRL-QAS, the PGR sampling framework from Wang et al. (2025), and the buffer transfer setup from Kundu & Feld (2026). Also, the paper overlooks recent model-based quantum RL efforts, such as DreamQAS and MuZero circuit synthesis.

  • Theorems 3.1 and 3.2 assume unguided Haar-random exploration over the full unitary space. In practice, GenQAS searches exclusively within a local neighborhood anchored by an MPS initialization already within trace distance δ\delta of the ground state (Lemma 3.4), meaning the global bounds do not model the localized MDP being solved.

  • Performance is volatile with respect to Generation Ratio GrG_r. A high synthetic ratio (Gr=0.8G_r=0.8) degrades success probability to 0.28 at 6 qubits (vs. 0.60 baseline), yet is required at 12 qubits to avoid collapse (0.79 vs. 0.38 for Gr=0.4G_r=0.4). The absence of an adaptive gating mechanism limits practical deployment.

  • Evaluated entirely with noiseless state-vector simulation. The transition model's stability under realistic measurement shot noise is quite meaningful as QAS would naturally address the noise distribution changes.

Questions for authors
  • How does generative replay perform on standard RL-QAS from scratch (∣0⟩⊗N\vert{}0\rangle^{\otimes N}) without the MPS warm start?

  • How does this 1-step MLP model compare in training cost and policy stability to recurrent world-model frameworks like DreamQAS? I'd love to see more systematic baseline comparison, not just TensorRL-QAS.

  • Can GrG_r be adjusted dynamically during training using transition-model validation error to eliminate manual per-system sweeps?

  • Does the learned transition and reward predictor fψf_\psi maintain accuracy when trained on noisy expectation values from finite measurement shots (10310^3–10410^4)?

6 comments
AuthorsOpenReview ↗

Brief response to reviewer comments

W →\rightarrow Weakness; Q →\rightarrow Questions. All the changes are highlighted in red in the revised manuscript.

(W1 and Q1) The ablation study is largely restricted...

We sincerely thank the reviewer for this valuable suggestion. We evaluated generative replay (Gr=0.4G_r=0.4) on cold-start QAS (∣0⟩⊗N|0\rangle^{\otimes N}) across all molecules:

SystemSettingSuccessMin Err (Ha)DepthGates (CX/RY/RZ)Time/Ep
6q BeH2\mathrm{BeH}_2GenQAS (wo TN)78.6%3.1×10−53.1 \times 10^{-5}1429 (11/10/8)17.65s
GenQAS (with TN)88.8%7.2×10−57.2 \times 10^{-5}714 (6/5/3)2.38s
8q H2O\mathrm{H}_2\mathrm{O}GenQAS (wo TN)69.0%1.1×10−31.1 \times 10^{-3}4186 (67/11/8)47.00s
GenQAS (with TN)84.6%1.2×10−31.2 \times 10^{-3}615 (5/8/2)2.41s
10q H2O\mathrm{H}_2\mathrm{O}GenQAS (wo TN)35.0%3.7×10−43.7 \times 10^{-4}2158 (30/23/5)147.50s
GenQAS (with TN)84.0%4.2×10−44.2 \times 10^{-4}79 (4/3/2)30.66s

Generative replay reaches chemical accuracy from scratch across all systems, while MPS warm starts prevent scalability collapse (84.0% vs. 35.0% at 10q), cut depth up to 6.8×6.8\times, and reduce episode time by 79.2–94.9%. Kindly check Section 4.5 of the revised manuscript.

(W2.1) The framework is a combination of three existing components.

We thank the reviewer and fully acknowledge that our pipeline builds upon foundational works. Our central contribution is identifying sample starvation in RL-QAS and showing that a simple 1-step model (fψf_\psi) mixed into updates counters it without complex world models or tree search. Furthermore, generative buffers transfer significantly more noise-resilient structure than passive replay (92.7% vs. 30.1% step reduction). Thanks to your suggestion, we also show GenQAS is resilient to measurement shot noise (W5, Q5).

(W2.2 and Q2) Comparison with DreamQAS...

We thank the reviewer for this constructive pointer. DreamQAS uses a recurrent GRU ensemble for multi-step imagined rollouts with real-VQE checks; MuZero (Ruiz et al., 2025) uses MCTS planning. GenQAS uses a 1-step feedforward MLP during updates with zero extra VQE calls:

  • 6-BeH2\mathrm{BeH}_2: DreamQAS: 0.058±<0.0010.058 \pm <0.001 mHa; GenQAS (Gr=0.4G_r=0.4): 0.049±0.0320.049 \pm 0.032 mHa.
  • 8-H2O\mathrm{H}_2\mathrm{O}: DreamQAS: 9.89±2.309.89 \pm 2.30 mHa; GenQAS (Gr=0.4G_r=0.4): 1.33±0.321.33 \pm 0.32 mHa (chemical accuracy).

Trade-offs: GenQAS has a larger parameter footprint (≈8.5M\approx 8.5\text{M} vs. 4.4M4.4\text{M}) and lacks adaptive gating. We updated Section 2 and provide an analysis in Section 4.3 and Appendix D.6.

(W3) Theorems 3.1 and 3.2 assume unguided Haar-random exploration...

We thank the reviewer for this insightful critique. In Section 3.3 and Appendix A.6, we add a Localization Lemma (Lemma 3.5): for anchored neighborhood Nδ(ρMPS)\mathcal{N}_\delta(\rho_{\mathrm{MPS}}), metric covering monotonicity yields C(ϵ,Nδ,∥⋅∥1)≤C(ϵ−2δ,U(2N),∥⋅∥1)\mathcal{C}(\epsilon, \mathcal{N}_\delta, \|\cdot\|_1) \le \mathcal{C}(\epsilon - 2\delta, \mathcal{U}(2^N), \|\cdot\|_1). Local sample complexity is mloc(ϵ,η)=O~(Cloc(ϵ)σ2ϵ2log⁡1η)m_{\mathrm{loc}}(\epsilon, \eta) = \widetilde{\mathcal{O}}(\frac{\mathcal{C}_{\mathrm{loc}}(\epsilon)\sigma^2}{\epsilon^2}\log \frac{1}{\eta}), so global Haar bounds act as an upper envelope. Because local dimension deff∼2N+1d_{\mathrm{eff}} \sim 2^{N+1} scales exponentially, transition sparsity persists at larger NN, necessitating generative replay.

(W4 and Q4) Performance is volatile with respect to GrG_r..

We thank the reviewer for this constructive observation. In Sections 4.1 and 5, we highlight that at 6q, Gr=0.4G_r=0.4 achieved 0.88 success (vs. 0.60 baseline), while Gr=0.8G_r=0.8 degraded to 0.28 due to model bias. At 12q, this inverts (Gr=0.8G_r=0.8 reaches 0.79 vs. 0.38 for Gr=0.4G_r=0.4) due to severe starvation. We acknowledge our probe heuristic is manual; automated uncertainty-aware gating (e.g., scaling GrG_r via validation loss or ensemble disagreement) is vital for robust deployment.

(W5 and Q5) Evaluated entirely with noiseless state-vector simulation.

We thank the reviewer for highlighting this practical aspect. We evaluated GenQAS (with Gr=0.4G_r=0.4) under finite shot budgets (103,5×103,10410^3, 5\times10^3, 10^4):

Shots6-BeH2BeH_2 Err / D8-H2OH_2O Err / D10-H2OH_2O Err / DSuccess
10310^35.4×10−55.4\times 10^{-5} / 14.8×10−54.8\times 10^{-5} / 44.8×10−54.8\times 10^{-5} / 2100%
5×1035\times10^31.7×10−51.7\times 10^{-5} / 22.2×10−42.2\times 10^{-4} / 22.2×10−42.2\times 10^{-4} / 3100%
10410^44.5×10−54.5\times 10^{-5} / 43.1×10−63.1\times 10^{-6} / 53.1×10−63.1\times 10^{-6} / 5100%

D is Depth. All runs achieve chemical accuracy without error drift or increasing depth of ansatz with stable runtime of 3.15-3.64 s/episode. These are added to Section 4 and Appendix D.7 in revised manuscript

AuthorsOpenReview ↗

Official Comment By Authors

Dear Reviewer,

Thank you again for your review, and we fully understand that the discussion period comes on top of many other commitments.

We have posted a detailed response to each of your points, with the additional experiments you asked for. As the discussion period is coming to a close, we would be grateful to hear whether our response addresses your concerns. If it does, we would appreciate it if you could consider updating your score. If any point still seems unresolved, please let us know, we are glad to clarify or provide further clarification before the discussion ends.

Best regards,

The Authors

I appreciate the authors for running the additional experiments (cold-start ablations, DreamQAS comparison, and finite-shot runs) and moderating Theorem 3.5 to Proposition 3.6. However, I am maintaining my score of weak rejection mainly due to the following two concerns. Maybe I'm somewhat more leaning to pessimistic side. 1) the core mechanism remains unresolved: generating 1-step transitions strictly from already-stored (s,a)(s,a) pairs does not demonstrate expanded state-action coverage, leaving the distinction between sample-starvation mitigation and target smoothing as an unverified hypothesis. 2) the framework remains a combination of existing techniques (TensorRL-QAS, PGR, and buffer transfer) that still requires manual per-system tuning of a sensitive GrG_r hyperparameter without adaptive gating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A brief summary of the rebuttal revisions

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and all three reviewers for your service and engaging with our work. Your thorough and constructive feedback throughout the discussion period helped us not only strengthen the technical claims of the paper but improve the flow. In the following we provide a unified summary of the new experiments, theoretical claims, and manuscript revisions made in response to all reviewers:

1. Numerical baselines and controls

  • Cold start ablation and passive baseline (cEHf, wWxS): We evaluated cold start GenQAS without the MPS warm start i.e. starting from ∣0⟩⊗N|0\rangle^{\otimes N} and cold start passive PER in Section 4.5 and Appendix Table 8. Compared to cold start passive PER, cold start GenQAS shows improvement specifically increasing chemical accuracy (C.A.) success from 33.3% to 78.6% on 6-BeH2\mathrm{BeH}_2; reducing episode time by 39.6%; to 17.65s from 29.20s and from 25.3% to 69.0% on 8-H2O\mathrm{H}_2\mathrm{O};reducing time by 18.0% to 47.00s from 57.30s. Adding the MPS warm start further reduces episode time by 86.5%-94.9% and prevents scalability collapse at 10-qubit (84.0% vs. 35.0%).

  • DreamQAS comparison (cEHf): We investigated a direct baseline comparison against DreamQAS in Sections 2, 4.3, and Appendix D.6. GenQAS achieves lower mean error on 6-BeH$_2$ i.e. 0.049±0.0320.049 \pm 0.032 mHa vs. 0.058±<0.0010.058 \pm <0.001 mHa. Meanwhile reaches chemical accuracy on 8-H$_2$O i.e. 1.33±0.321.33 \pm 0.32 mHa vs. 9.89±2.309.89 \pm 2.30 mHa without extra VQE evaluations. We also acknowledge GenQAS's larger parameter footprint.

  • Prioritized Duplicate Replay control (wWxS, sj4h): We evaluated Duplicate PER at matched Gr=0.4G_r=0.4 on 8-H$_2$O in Section 4.3, Appendix D.10. Our investigation reveal that duplicating real samples failed to reach C.A. across 2 seeds whereas GenQAS reached 1.191.19 mHa with 84.5% success, with depth 4 circuit containing 3 CNOTs.

  • Noisy-reward control (wWxS): Here we conducted a 3 seed noisy-reward control on 8-H$_2$O under noiseless simulation, perturbing only replay rewards with Gaussian noise which is scaled to the reward predictors training RMSE. The investigation show that control achieved C.A. in 2/3 seeds with minimum errors 1.7251×10−31.7251\times 10^{-3}, 1.1879×10−31.1879\times 10^{-3}, 1.2066×10−31.2066\times 10^{-3} Ha versus 3/3 for GenQAS.

  • Finite shot noise evaluation (cEHf): We added a finite shot evaluations with 103,5×103,10410^3, 5\times 10^3, 10^4 shots across 6-, 8-, and 10-qubit systems in Section 4 and Appendix D.7, demonstrating C.A. convergence without a significant increase in quantum circuit size across tasks.

2. Theoretical claims and framing of mechanism

  • Matched warm start theory and localization (cEHf, wWxS, sj4h): In the revision we downgraded Theorem 3.5 to Proposition 3.6 in Section 3.3 and Appendix A.5 (contains the proof) under matched MPS warm starts. Moreover we unconditioned fψf_\psi, and added a Localization Lemma: Lemma 3.5, Appendix A.6. Where we explicitly clarify that Proposition 3.6 serves as conditional analytical motivation rather than an unconstrained guarantee.

  • Mechanism and GrG_r tuning (cEHf, wWxS, sj4h): We revised our earlier wording to note that neither the duplication control nor the buffer transfer result establishes sample starvation mitigation as the sole mechanism. Accordingly, the interplay between representation smoothing and starvation mitigation remains a hypothesis. We also acknowledged manual GrG_r tuning as a usability limitation and outlined an adaptive scheduling formulation grounded in Proposition 3.6 as future work.

3. Reporting corrections and clarification of resources

  • 12-Qubit reporting (wWxS): Replaced the peak run claim in the Abstract with the 5 seed mean improvement of 6.4×6.4\times under the simulated annealing benchmark threshold 2.5×10−22.5\times 10^{-2} Ha. Explicitly noting that no evaluated method reaches C.A. for 12-qubit. We also added 5 seed mean ±\pm CI statistics with validation selected Gr∗G_r^* in Section 4.1.

  • Full executable circuit costs (sj4h): We retained Table 1 for suffix-only RL comparisons, and expanded Section 4.3 and Appendix B.6 to report complete executable circuit costs i.e. U=VtUTNU = V_t U_{TN}, including the fixed MPS prefix from Appendix Table 3; against baselines such as UCCSD.

  • Table correction and context clarification (wWxS): We corrected the indexing error in Table 9 i.e. 10-H$_2$O shot noise rows: 9.0×10−59.0\times 10^{-5}, 6.6×10−56.6\times 10^{-5}, and 1.2×10−61.2\times 10^{-6} Ha and clarified that the 100% rate reflects terminal convergence on that test seed. We also added context for the narrow 5 seed confidence intervals in Table 4 in Appendix D.1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.