- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The paper introduces a surrogate-assisted actor–critic approach for hybrid quantum reinforcement learning, where a classical neural surrogate is fitted to PQC input-output pairs from the replay buffer and replaces costly quantum-circuit queries during actor and critic-target updates, while quantum measurements are retained during environment interaction and evaluation. The approach is evaluated on four MuJoCo continuous-control tasks, with extensive experiments on Humanoid covering classical/stochastic baselines, surrogate and circuit capacity, shot counts, and update strategies. The paper additionally provides theoretical bounds separating surrogate-fitting and mean-feature relaxation errors and analyzes when surrogate gradients approximate the sampled quantum-policy objective.
Critical strengths
Thorough, statistically rigorous, and transparent experimental evaluation.
The paper evaluates the approach across multiple continuous-control environments and uses substantial seed cohorts, classical and stochastic controls. The authors are also unusually candid about experimental limitations, explicitly distinguishing inconclusive, exploratory, and post-hoc results rather than overstating the evidence.
Careful theoretical analysis of surrogate-assisted training.
The paper separates surrogate-fitting error from the bias introduced by the mean-feature relaxation and analyzes when surrogate gradients approximate the sampled quantum-policy objective. Importantly, it also identifies cases where even an accurate surrogate can produce misleading gradients, providing useful insight into the limitations of the proposed approach.
Critical weaknesses
Insufficient motivation, positioning, and methodological overview.
The paper does not establish why a PQC should be used in the first place or what potential advantage it offers over classical alternatives. This is compounded by limited related-work coverage: recent quantum RL approaches for continuous action spaces [1,2], classical deployment of quantum-trained policies [3], and classically trainable quantum models (such as IQPs [4], tensor networks [5], Pauli path propagation [6]) are not adequately discussed. Given that the learned surrogate itself can replace the PQC at deployment, clearer motivation in terms of expressivity, Fourier structure, or another plausible route to quantum advantage is important. Moreover, the detailed architecture and training-flow diagrams are relegated to the appendix, although they substantially clarify the interaction between PQC, surrogate, actor, critic, and replay buffer. Including such an overview in the main paper would make the method considerably easier to understand.
The experimental setup does not sufficiently establish an advantage of the proposed quantum approach.
Although the statistical methodology is unusually thorough, the baselines largely replace individual components within the authors' pipeline rather than providing broader end-to-end comparisons against strong classical RL and alternative PQC-based methods. In particular, the benefit of the PQC over an appropriately structured classical model is not isolated. Recent dequantization results [7,8] demonstrate that classical Fourier-based models can reproduce the performance of certain QNNs and quantum kernels, making it important to compare against classical models reflecting the PQC's inductive/Fourier structure rather than only generic neural and random-feature baselines. Scalability beyond 5–10 qubits also remains unclear, as does the sensitivity to important surrogate-training choices such as fitting steps, parameter retention, and target-update strategy.
[1] Wu, Shaojun, et al. "Quantum reinforcement learning in continuous action space." Quantum 9 (2025): 1660.
[2] Sinha, Akash, Antonio Macaluso, and Matthias Klusch. "Nav-Q: quantum deep reinforcement learning for collision-free navigation of self-driving cars." Quantum Machine Intelligence 7.1 (2025): 19.
[3] Ngo, Quynh Tu, et al. "Quantum Reinforcement Learning with Classical Policy Deployment for Resource Allocation in Multi-Beam GEO-LEO Satellite Networks." IEEE Internet of Things Journal (2026).
[4] Recio-Armengol, Erik, Shahnawaz Ahmed, and Joseph Bowles. "Train on classical, deploy on quantum: scaling generative quantum machine learning to a thousand qubits." arXiv preprint arXiv:2503.02934 (2025).
[5] Liao, Hai-Jun, et al. "Differentiable programming tensor networks." Physical Review X 9.3 (2019): 031041.
[6] Angrisani, Armando, et al. "Simulating quantum circuits with arbitrary local noise using pauli propagation." PRX Quantum 7.2 (2026): 020313.
[7] Sahebi, Mehrad, et al. "On dequantization of supervised quantum machine learning via random Fourier features." arXiv preprint arXiv:2505.15902 (2025).
[8] Rodriguez-Grasa, Pablo, et al. "Towards Surrogate Based Dequantization of Quantum Reinforcement Learning." arXiv preprint arXiv:2609.16266 (2026).
Questions for authors
-
Motivation and positioning of the quantum component (clarification). What property of the particular PQC used here is expected to provide an advantage over classical representations for these RL tasks? In particular, how do you position the approach relative to recent work on quantum RL for continuous control [1,2], classical deployment of quantum-trained policies [3], classically trainable quantum models [4-6], and recent dequantization results showing that classical Fourier-based models can reproduce the performance of certain quantum learning models [7,8]? A clearer route from the chosen PQC to a potential quantum benefit would materially strengthen the motivation.
-
Isolating the benefit of the PQC (additional evidence). The current controls establish that the PQC pipeline can outperform the tested neural and stochastic baselines, but do not clearly establish that this advantage is specific to the quantum model. This is particularly interesting given that the learned surrogate can replace the PQC at deployment while retaining the high-return seeds, and the classical stochastic bottleneck is not distribution/entropy-matched to the PQC. Could you provide evidence against a stronger classical baseline that reproduces relevant properties of the PQC, e.g. its Fourier/inductive structure or output statistics? This would help determine whether the observed advantage originates from the PQC or from the particular stochastic/nonlinear training channel.
-
Scalability with the number of qubits (clarification). The experiments consider primarily 5- and 10-qubit configurations, with the smaller configurations sometimes performing equally well or better. How do you expect the surrogate architecture, fitting cost, and required amount of replay data to scale with increasing qubit count? In particular, is there empirical or theoretical evidence that a fixed/polynomially growing surrogate remains sufficient as the PQC becomes larger, rather than eventually requiring exponentially increasing capacity?
-
Sensitivity to surrogate-training choices (additional evidence). The proposed method relies on repeatedly fitting the surrogate using 32 Adam steps on replay minibatches, retaining its parameters between fits and immediately hard-copying the surrogate target. How sensitive are the final policy performance and gradient quality to these choices (e.g., number of fitting steps, surrogate update frequency, parameter retention/reinitialization, or target-network update strategy)? An ablation would help establish whether the reported performance is robust or depends strongly on the particular surrogate-training configuration.
2 comments
Thank you for the detailed rebuttal and the transparent discussion of the paper's limitations.
I agree that demonstrating quantum advantage is not necessary for the proposed contribution. The reduction in circuit queries during surrogate-assisted training is meaningful, and the clarification of the paper's scope is helpful.
However, my main concerns remain. In particular, the contribution of the quantum component during training is not isolated, especially given the strong performance retained when replacing its outputs at deployment. The absence of Fourier- or output-distribution-matched classical baselines leaves this question unresolved. Furthermore, the provided complexity expressions do not establish scalability of surrogate accuracy or sample requirements, and the sensitivity of final policy performance to surrogate-training choices remains insufficiently evaluated.
I appreciate that the authors explicitly acknowledge these limitations and do not overstate their conclusions. Nevertheless, I will maintain my overall recommendation of weak rejection, while recognizing that the theoretical results are more appropriately understood as conditional guarantees than as evidence of a fundamental soundness problem.
We thank the reviewer for their careful reading and insightful questions.
On the weaknesses:
Our motivation is to train measured hybrid policies without repeatedly querying the circuit during minibatch updates. Figure 1 already separates interaction/evaluation, replay fitting, and actor/critic-target updates. The introduction explicitly distinguishes this training question from establishing a quantum advantage and states that the mechanism behind the observed differences in learning outcomes remains unidentified. Appendix A provides the detailed architectures. See response to Q1 for more details.
Our controls compare representations and training procedures within a shared actor–critic framework. On Humanoid, the hybrid policy achieves terminal returns of at least 4,000 in 10/15 runs (66.7%), compared with 4/15 (26.7%) for each learned classical baseline, FC and FCM. Figure 3 presents the full terminal-threshold profile. These results suggest more frequent high-return outcomes under the reported hybrid training procedure.
Table 3 also reports 2.05M versus 650.38M channel invocations at the same 1.7M-interaction budget, but these results indeed do not establish superiority over classical RL generally (cf. Section 3.2.2).
On the questions:
The motivation is the training bottleneck created by measured quantum components: actor and critic-target updates can require a large number of circuit calls. We deliberately study how to reduce this cost using surrogate-assisted hybrid policies. The chosen PQC is a concrete test case for the procedure. A circuit-specific advantage over classical representations is neither an assumption of the method nor an objective of this study.
Relative to the quantum RL approaches in [1,2], our contribution is replay-based response fitting for actor and critic-target updates. The classical deployment approach in [3] uses a policy translator, whereas our deployment experiment reuses the surrogate fitted during training, without additional fitting. Methods [4–6] exploit particular model structures and assumptions for classical training or simulation while our surrogate is fitted from observed circuit control–output pairs. The dequantization results in [7,8] concern supervised quantum models and kernelized fitted Q-iteration under specified assumptions, respectively. These works provide relevant context for the choice and training of quantum models, and we will expand the related-work discussion to clarify these distinctions.
Our central question is whether a classical surrogate can replace PQC queries during training while retaining effective policy learning. The PQC is the component being approximated and the contribution concerns the surrogate-assisted training procedure.
Section 3.4 directly compares surrogate-forward and sampled-forward updates while retaining quantum measurements during interaction and evaluation. Appendix E additionally examines replacement at deployment: using the stored surrogate without further fitting retains all ten high-return Humanoid seeds. The theoretical analysis complements these experiments by bounding the errors introduced by surrogate fitting and mean-feature substitution.
FCM provides a comparison with a matched classical architecture and CSB uses a classical stochastic channel with the same surrogate-training protocol as PQC. We did not include Fourier- or output-law-matched baselines because they address the source of the performance differences, whereas our study evaluates surrogate-assisted training.
For input dimension d=(2N+2)M and hidden width h, Appendix B.6 gives the parameter count (d+N+1)h+N and fitting cost O(Kfitbh(d+N)), where Kfit and b denote fitting steps and sub-batch size.
The algorithm does not prescribe exponential width growth. However, these architecture costs do not guarantee that sufficient capacity or replay coverage remains polynomial. Our conclusions are restricted to the tested configurations. The compact-model results do not establish large-qubit scalability.
Sections 2.2–2.3 and Algorithm 1 specify the persistent 32-step fitting protocol and immediate surrogate-target copy. Appendix B.7 explores fitting budgets and persistence in synthetic probes, and Appendices D.3–D.4 report alternative fitting configurations and live response-sensitivity diagnostics.
These studies do not isolate how the fitting schedule, parameter retention, or target-update strategy affects final policy performance. Theorem 1 likewise requires both response and Jacobian accuracy and does not infer gradient fidelity from BCE fitting alone or establish convergence.
We thank the reviewer again for the feedback and suggestions. We hope these clarifications are helpful in the final assessment. We will strengthen the positioning and related work discussion in the revised manuscript.