acceptodds
Under review as a conference paper at ICLR 2027

Training Hybrid Quantum Policies with Classical Surrogates for Actor–Critic Reinforcement Learning

Abstract

Reading out a parametrized quantum circuit through finite-shot measurements introduces a discrete stochastic component into hybrid quantum-classical neural networks, making gradient-based training inefficient. In this work, we introduce a surrogate-assisted actor-critic method for training hybrid quantum policies that avoids differentiating the quantum circuit during backpropagation. A classical neural surrogate trained on the circuit's input-output pairs from a replay buffer provides a differentiable substitute for actor updates and critic-target computation, while forward interaction and evaluation retain the quantum circuit's measurements. Our experiments on standard MuJoCo continuous-control tasks demonstrate promising policy learning. Notably, on Humanoid, more runs reach high returns than with learned classical baselines. Moreover, the best average return in our ablation comes from a smaller circuit / surrogate, suggesting that strong performance does not depend on a large network. We also derive conditional upstream gradient-error bounds that separate surrogate fitting from the mean-feature relaxation. Together, the results support surrogate-assisted training as an effective and efficient approach to learning hybrid quantum policies.

Primary area:
reinforcement learning
Keywords:
Quantum Machine Learning, Deep Reinforcement Learning, Parametrized Quantum Circuits, Hybrid Quantum-Classical Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_rhoROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper introduces a surrogate-assisted actor–critic approach for hybrid quantum reinforcement learning, where a classical neural surrogate is fitted to PQC input-output pairs from the replay buffer and replaces costly quantum-circuit queries during actor and critic-target updates, while quantum measurements are retained during environment interaction and evaluation. The approach is evaluated on four MuJoCo continuous-control tasks, with extensive experiments on Humanoid covering classical/stochastic baselines, surrogate and circuit capacity, shot counts, and update strategies. The paper additionally provides theoretical bounds separating surrogate-fitting and mean-feature relaxation errors and analyzes when surrogate gradients approximate the sampled quantum-policy objective.

Critical strengths

Thorough, statistically rigorous, and transparent experimental evaluation.

The paper evaluates the approach across multiple continuous-control environments and uses substantial seed cohorts, classical and stochastic controls. The authors are also unusually candid about experimental limitations, explicitly distinguishing inconclusive, exploratory, and post-hoc results rather than overstating the evidence.

Careful theoretical analysis of surrogate-assisted training.

The paper separates surrogate-fitting error from the bias introduced by the mean-feature relaxation and analyzes when surrogate gradients approximate the sampled quantum-policy objective. Importantly, it also identifies cases where even an accurate surrogate can produce misleading gradients, providing useful insight into the limitations of the proposed approach.

Critical weaknesses

Insufficient motivation, positioning, and methodological overview.

The paper does not establish why a PQC should be used in the first place or what potential advantage it offers over classical alternatives. This is compounded by limited related-work coverage: recent quantum RL approaches for continuous action spaces [1,2], classical deployment of quantum-trained policies [3], and classically trainable quantum models (such as IQPs [4], tensor networks [5], Pauli path propagation [6]) are not adequately discussed. Given that the learned surrogate itself can replace the PQC at deployment, clearer motivation in terms of expressivity, Fourier structure, or another plausible route to quantum advantage is important. Moreover, the detailed architecture and training-flow diagrams are relegated to the appendix, although they substantially clarify the interaction between PQC, surrogate, actor, critic, and replay buffer. Including such an overview in the main paper would make the method considerably easier to understand.

The experimental setup does not sufficiently establish an advantage of the proposed quantum approach.

Although the statistical methodology is unusually thorough, the baselines largely replace individual components within the authors' pipeline rather than providing broader end-to-end comparisons against strong classical RL and alternative PQC-based methods. In particular, the benefit of the PQC over an appropriately structured classical model is not isolated. Recent dequantization results [7,8] demonstrate that classical Fourier-based models can reproduce the performance of certain QNNs and quantum kernels, making it important to compare against classical models reflecting the PQC's inductive/Fourier structure rather than only generic neural and random-feature baselines. Scalability beyond 5–10 qubits also remains unclear, as does the sensitivity to important surrogate-training choices such as fitting steps, parameter retention, and target-update strategy.

[1] Wu, Shaojun, et al. "Quantum reinforcement learning in continuous action space." Quantum 9 (2025): 1660.

[2] Sinha, Akash, Antonio Macaluso, and Matthias Klusch. "Nav-Q: quantum deep reinforcement learning for collision-free navigation of self-driving cars." Quantum Machine Intelligence 7.1 (2025): 19.

[3] Ngo, Quynh Tu, et al. "Quantum Reinforcement Learning with Classical Policy Deployment for Resource Allocation in Multi-Beam GEO-LEO Satellite Networks." IEEE Internet of Things Journal (2026).

[4] Recio-Armengol, Erik, Shahnawaz Ahmed, and Joseph Bowles. "Train on classical, deploy on quantum: scaling generative quantum machine learning to a thousand qubits." arXiv preprint arXiv:2503.02934 (2025).

[5] Liao, Hai-Jun, et al. "Differentiable programming tensor networks." Physical Review X 9.3 (2019): 031041.

[6] Angrisani, Armando, et al. "Simulating quantum circuits with arbitrary local noise using pauli propagation." PRX Quantum 7.2 (2026): 020313.

[7] Sahebi, Mehrad, et al. "On dequantization of supervised quantum machine learning via random Fourier features." arXiv preprint arXiv:2505.15902 (2025).

[8] Rodriguez-Grasa, Pablo, et al. "Towards Surrogate Based Dequantization of Quantum Reinforcement Learning." arXiv preprint arXiv:2609.16266 (2026).

Questions for authors
  1. Motivation and positioning of the quantum component (clarification). What property of the particular PQC used here is expected to provide an advantage over classical representations for these RL tasks? In particular, how do you position the approach relative to recent work on quantum RL for continuous control [1,2], classical deployment of quantum-trained policies [3], classically trainable quantum models [4-6], and recent dequantization results showing that classical Fourier-based models can reproduce the performance of certain quantum learning models [7,8]? A clearer route from the chosen PQC to a potential quantum benefit would materially strengthen the motivation.

  2. Isolating the benefit of the PQC (additional evidence). The current controls establish that the PQC pipeline can outperform the tested neural and stochastic baselines, but do not clearly establish that this advantage is specific to the quantum model. This is particularly interesting given that the learned surrogate can replace the PQC at deployment while retaining the high-return seeds, and the classical stochastic bottleneck is not distribution/entropy-matched to the PQC. Could you provide evidence against a stronger classical baseline that reproduces relevant properties of the PQC, e.g. its Fourier/inductive structure or output statistics? This would help determine whether the observed advantage originates from the PQC or from the particular stochastic/nonlinear training channel.

  3. Scalability with the number of qubits (clarification). The experiments consider primarily 5- and 10-qubit configurations, with the smaller configurations sometimes performing equally well or better. How do you expect the surrogate architecture, fitting cost, and required amount of replay data to scale with increasing qubit count? In particular, is there empirical or theoretical evidence that a fixed/polynomially growing surrogate remains sufficient as the PQC becomes larger, rather than eventually requiring exponentially increasing capacity?

  4. Sensitivity to surrogate-training choices (additional evidence). The proposed method relies on repeatedly fitting the surrogate using 32 Adam steps on replay minibatches, retaining its parameters between fits and immediately hard-copying the surrogate target. How sensitive are the final policy performance and gradient quality to these choices (e.g., number of fitting steps, surrogate update frequency, parameter retention/reinitialization, or target-network update strategy)? An ablation would help establish whether the reported performance is robust or depends strongly on the particular surrogate-training configuration.

2 comments

We thank the reviewer for their careful reading and insightful questions.

On the weaknesses:

  1. Our motivation is to train measured hybrid policies without repeatedly querying the circuit during minibatch updates. Figure 1 already separates interaction/evaluation, replay fitting, and actor/critic-target updates. The introduction explicitly distinguishes this training question from establishing a quantum advantage and states that the mechanism behind the observed differences in learning outcomes remains unidentified. Appendix A provides the detailed architectures. See response to Q1 for more details.

  2. Our controls compare representations and training procedures within a shared actor–critic framework. On Humanoid, the hybrid policy achieves terminal returns of at least 4,000 in 10/15 runs (66.7%), compared with 4/15 (26.7%) for each learned classical baseline, FC and FCM. Figure 3 presents the full terminal-threshold profile. These results suggest more frequent high-return outcomes under the reported hybrid training procedure.
    Table 3 also reports 2.05M versus 650.38M channel invocations at the same 1.7M-interaction budget, but these results indeed do not establish superiority over classical RL generally (cf. Section 3.2.2).

On the questions:

  1. The motivation is the training bottleneck created by measured quantum components: actor and critic-target updates can require a large number of circuit calls. We deliberately study how to reduce this cost using surrogate-assisted hybrid policies. The chosen PQC is a concrete test case for the procedure. A circuit-specific advantage over classical representations is neither an assumption of the method nor an objective of this study.
    Relative to the quantum RL approaches in [1,2], our contribution is replay-based response fitting for actor and critic-target updates. The classical deployment approach in [3] uses a policy translator, whereas our deployment experiment reuses the surrogate fitted during training, without additional fitting. Methods [4–6] exploit particular model structures and assumptions for classical training or simulation while our surrogate is fitted from observed circuit control–output pairs. The dequantization results in [7,8] concern supervised quantum models and kernelized fitted Q-iteration under specified assumptions, respectively. These works provide relevant context for the choice and training of quantum models, and we will expand the related-work discussion to clarify these distinctions.

  2. Our central question is whether a classical surrogate can replace PQC queries during training while retaining effective policy learning. The PQC is the component being approximated and the contribution concerns the surrogate-assisted training procedure.
    Section 3.4 directly compares surrogate-forward and sampled-forward updates while retaining quantum measurements during interaction and evaluation. Appendix E additionally examines replacement at deployment: using the stored surrogate without further fitting retains all ten high-return Humanoid seeds. The theoretical analysis complements these experiments by bounding the errors introduced by surrogate fitting and mean-feature substitution.
    FCM provides a comparison with a matched classical architecture and CSB uses a classical stochastic channel with the same surrogate-training protocol as PQC. We did not include Fourier- or output-law-matched baselines because they address the source of the performance differences, whereas our study evaluates surrogate-assisted training.

  3. For input dimension d=(2N+2)Md=(2N+2)M and hidden width hh, Appendix B.6 gives the parameter count (d+N+1)h+N(d+N+1)h+N and fitting cost O(Kfitbh(d+N)),O\left(K_{\mathrm{fit}}bh(d+N)\right), where KfitK_{\mathrm{fit}} and bb denote fitting steps and sub-batch size.
    The algorithm does not prescribe exponential width growth. However, these architecture costs do not guarantee that sufficient capacity or replay coverage remains polynomial. Our conclusions are restricted to the tested configurations. The compact-model results do not establish large-qubit scalability.

  4. Sections 2.2–2.3 and Algorithm 1 specify the persistent 32-step fitting protocol and immediate surrogate-target copy. Appendix B.7 explores fitting budgets and persistence in synthetic probes, and Appendices D.3–D.4 report alternative fitting configurations and live response-sensitivity diagnostics.
    These studies do not isolate how the fitting schedule, parameter retention, or target-update strategy affects final policy performance. Theorem 1 likewise requires both response and Jacobian accuracy and does not infer gradient fidelity from BCE fitting alone or establish convergence.

We thank the reviewer again for the feedback and suggestions. We hope these clarifications are helpful in the final assessment. We will strengthen the positioning and related work discussion in the revised manuscript.

Thank you for the detailed rebuttal and the transparent discussion of the paper's limitations.

I agree that demonstrating quantum advantage is not necessary for the proposed contribution. The reduction in circuit queries during surrogate-assisted training is meaningful, and the clarification of the paper's scope is helpful.

However, my main concerns remain. In particular, the contribution of the quantum component during training is not isolated, especially given the strong performance retained when replacing its outputs at deployment. The absence of Fourier- or output-distribution-matched classical baselines leaves this question unresolved. Furthermore, the provided complexity expressions do not establish scalability of surrogate accuracy or sample requirements, and the sensitivity of final policy performance to surrogate-training choices remains insufficiently evaluated.

I appreciate that the authors explicitly acknowledge these limitations and do not overstate their conclusions. Nevertheless, I will maintain my overall recommendation of weak rejection, while recognizing that the theoretical results are more appropriately understood as conditional guarantees than as evidence of a fundamental soundness problem.

Reviewer_5hKgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper uses a classical surrogate to train a parameterized quantum circuit controller. The PQC serves as a bitstring sampler, taking input from a pre-network that receives the state information. The resulting bitstring and bypass information are then fed into a post-network to generate actions in the RL process. The results show good performance.

Critical strengths
  1. The proposed method eliminates the need to calculate gradients through the quantum device, which would require extensive access to the quantum device.
  2. Theoretical derivations support the idea, and the empirical experiments demonstrate strong results compared with other baselines.
Critical weaknesses
  1. In Table I, “IQM” appears without explanation and is not defined until page 24 in the appendix. This makes the paper harder to follow, as readers must look far ahead to find the definition.
  2. Fig. 6 is somewhat confusing because its symbols differ from those used in the main text.
  3. It is unclear why using a quantum sampler in the forward pass yields higher rewards on average. The classical sampling baseline appears to use a very simple model compared with modern architectures such as transformers.
Questions for authors
  1. Have you tried using more modern network architectures, such as transformers, as the sampler baseline?
  2. Since the quantum policy uses quantum sampling to provide information for actions, have you investigated its performance on real quantum hardware?
  3. How does quantum noise, such as depolarizing noise, affect the sampling process?
1 comment

We thank the reviewer for their careful reading and constructive questions.

On the critical weaknesses:

  1. IQM denotes the interquartile mean. Under our finite-sample convention, we remove the lowest and highest ⌊n/4⌋\lfloor n/4\rfloor seed-level scores and average the remainder. Appendix D.1 gives the formula. We will define the acronym and convention at their first appearance in Table 1 in the revised version.

  2. The correspondence between the diagram labels and main-text notation is stated above Figure 6 in Appendix A: qi=u,qo=B,qo,pred=fϕ(u).q_i=u,\qquad q_o=B,\qquad q_{o,\mathrm{pred}}=f_\phi(u). The direct pre-network–post-network connection represents the bypass (c). We will use the same notation in the main text and figures in the revised version to make these connections easier to follow.

  3. As noted in the introduction and discussion, we have not focused on what causes the difference in Humanoid outcomes. The experiments demonstrate that the surrogate-assisted procedure can train useful hybrid policies, but do not establish which property of the quantum channel causes the observed difference.
    The classical comparisons include both learned bottlenecks and a stochastic channel. In particular, FCM includes the same width-2048 hidden layer and parameter count as the surrogate and is trained directly through the actor objective. CSB instead uses the same surrogate-assisted training protocol and complete-bitstring shot aggregation as PQC (cf. Appendix C.3). Their purpose is to assess the feasibility and performance of surrogate-assisted hybrid-policy training under comparable architectures and settings.

On the questions:

  1. We did not include a transformer-based sampler (or other more modern architectures) because we tested whether surrogate-assisted training can produce useful hybrid quantum policies and how those policies compare with classical alternatives under comparable architectures and training settings.
    Introducing a transformer-based sampler would change the bottleneck architecture and its inductive bias. Such an experiment would assess whether a different architecture is better suited to the task, requiring its own design, tuning, and capacity and computational-budget comparisons. This would broaden the study beyond the controlled comparisons considered here.

  2. All reported experiments use simulated circuits, as stated in Section 3. The submission contains no physical-hardware evaluation.
    The measured query reduction comes from removing circuit calls from actor and critic-target minibatch updates while retaining them during interaction and evaluation. The reported process-time comparison is an observed simulator result without hardware normalization, and we do not present it as a hardware runtime advantage.

  3. The reported circuits already include both bit-flip and phase-flip noise, each with probability 10−310^{-3} per U3 gate (Appendix C.1). We will make this setting explicit in the main experimental description.
    Noise changes the circuit’s measurement distribution and therefore the distribution of the modal bitstring supplied to the policy. Because the surrogate is trained on observed circuit responses, its fitting target incorporates the simulated noise. The results therefore demonstrate learning under the specified noisy-channel setting.
    The submitted experiments do not include a depolarizing-noise or noise-strength sweep. Consequently, they do not establish robustness across different noise models or strengths, and response fitting alone does not guarantee robustness to stronger noise or device drift.

We thank the reviewer again for the constructive feedback. We hope these answers help clarify the method and experiments for the final assessment. We will incorporate the proposed improvements into the revised manuscript.

Comments on the paper

1 comment

We thank the reviewer for their careful reading and insightful questions.

On the critical weaknesses:

  1. The training contribution is tested directly in §3.4: the surrogate removes circuit queries from minibatch updates while still learning useful policies, and this result does not require superiority over FC/FCM. The deployment controls address whether the circuit can be replaced in an already-trained actor but don't assess what caused successful learning. We will make these experimental purposes clearer, so the performance comparisons and deployment results are not interpreted as evidence of a quantum-specific advantage.

  2. The capacity and shot sweeps demonstrate successful training across several configurations, while their rankings are presented as exploratory. Changing the shot count changes the aggregated output law by construction, as explained in Section 3.3. The sweep therefore compares different finite-shot policies and is not interpreted as changing only the cost or precision of an otherwise fixed policy.
    The theoretical contribution is to quantify the approximation error and identify sufficient conditions for useful surrogate gradients. Theorem 1 bounds the propagated gradient error, while Corollary 1 provides a sufficient condition for the surrogate gradient to be a local ascent direction for the sampled objective with the critic held fixed.
    We discuss warm-up and checkpointing in our answer to Question 2.

On the questions:

  1. The CSB baseline uses the same surrogate-training protocol and mode aggregation, but the experiments do not include an output-law- or entropy-matched classical channel (cf. Appendix C.3). Appendix E.2 directly tests surrogate replacement within an already-trained policy. Surrogate features retain all ten high-return seeds without further fitting, while constant-half and fair-bit features retain nine. Keeping the learned bypass and network weights fixed is deliberate: the experiment evaluates replacing the deployed features without changing the rest of the trained actor.
    The surrogate-deployed actor also matches FCM’s architecture, allowing comparison between two training procedures with a common final architecture. The experiments do not isolate the bypass’s contribution during training or identify a quantum-specific training mechanism.

  2. Appendix C.5 documents the 1,001 warm-up pairs' inclusion, and Appendix B.1 restricts the population BCE result to genuine channel observations. Their small share of the replay buffer later in training does not show that their effect was negligible.
    New full-training runs are infeasible within the rebuttal period given their computational cost. We therefore cannot say whether excluding the placeholder pairs would change these results.
    Moreover, the saved checkpoints cannot reconstruct the historical shot streams exactly. The primary statistical analyses use training runs as their units and do not assume paired shot streams across architectures. Shared-prefix re-executions are reported separately and never counted as additional independent primary seeds (cf. Appendices D.1 and D.7).

  3. Appendix D.4, Table 14, already reports live directional response-error measurements at replay controls from three PQC runs, three CSB runs, and five derivative-target runs. Appendix D.3 additionally evaluates derivative-target fitting. These measurements partially address response sensitivity at visited controls, but indeed neither certify Theorem 1’s operator-norm assumption nor establish alignment through trained critics. Appendix B.7’s synthetic probes further demonstrate the distinction between response accuracy and derivative fidelity. For metric sensitivity, Figure 3 provides the complete terminal-threshold profile, while Appendix D reports late-training summaries and individual learning histories. We have not added seeds to the compact or low-shot experiments, and those comparisons remain exploratory.

We thank the reviewer again for the detailed feedback. We hope these answers clarify what our results show and help with the final assessment. We will incorporate the proposed clarifications into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.