- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the semiconductor quantum dot arrays optimization problem with multi-agent reinforcement learning. The authors propose using a factored representation of the action space, learned online, to decouple agents. The authore claim that the approach provides a scalable route toward rapid calibration of large-scale quantum processors.
Critical strengths
- The reviewer find the topic is well motivated. It is important to find a scalable way to calibrate quantum processors efficiently.
- The authors provide a good collection of ablation studies, including Kalman filter, various CNNs, IPPO, MAPPO, MADDPG, FACMAC...
- The authors demonstrated at least limited zero-shot scalability.
Critical weaknesses
- I think the authors should explain the requirements of using RL. If cross-capacitance matrix is got, then what is the reason of using a RL such as PPO?
- The reward and evaluations metric highly depend on the simulation software.
- There are some questions about the baseline fairness. (Detailed in the Questions Part.)
Questions for authors
- The authors hould clarify how to actually get those reward from actual devices if someone want to actually implement the algorithms.
- In Appendix C (Table 7), Bayesian optimization uses 19 Latin hypercube samples followed by only one GP-guided acquisition per window visit. Could the authors justify this configuration and evaluate whether allocating more evaluations to GP-guided optimization improves the baseline performance?
- Could the authors compare QADAPT against classical optimization or MARL baselines operating in the same virtualized action space? This would help distinguish the benefits of physics-informed preconditioning from those of the proposed multi-agent learning architecture.
RL benefits discussed, rewards explained, baseline configurations justified
We thank the reviewer for the opportunity to clarify our work. First, the cross-capacitance matrix is not available before tuning. It can only be estimated from CSDs that display charge transitions, which requires the device to already be partially tuned, and is why QADAPT estimates it online. Moreover, while an exact cross-capacitance matrix decouples the gates, it does not reveal where the target configuration lies. To illustrate the necessity of using RL, we have run an additional baseline in which the ground-truth cross-capacitance matrix is hypothetically known a priori, and applied from the start of each classical optimization round. The results are provided in Appendix D, and retain poor performance (see point 3). In our RL strategy, training across many simulated devices teaches QADAPT how the features in a CSD relate to the direction and distance to the target. Conversely, non-RL strategies reduce each measurement to a single scalar objective and must build their picture of the tuning landscape from scratch. In noise-dominated regions, such a scalar signal can be flat and give no indication of direction.
Second, the reward is used only during training. At deployment, QADAPT acts solely on measured CSDs and gate voltages, so no reward, and hence no simulator, is required on hardware. Since the reward depends only on the distance to a physically defined target configuration (now specified in a revised Appendix E), any simulator or digital twin that captures the same physics would yield the same reward. In Appendix I, we apply the same framework to superconducting-qubit calibration, where the simulator and reward is different simulator, showing that our action-factorization strategy extends beyond a single problem setting. Our contribution therefore does not hinge on a particular reward structure. The reward is a task-specific component that can be exchanged, whereas QADAPT’s online action-space factorization and role-shared decentralized learning remain unchanged across both settings.
BO is applied to sliding windows comprising two neighbouring plungers and their shared barrier (three voltages). For an array of N dots, this equates to N-1 windows. For the first visit to each window, 19 Latin hypercube samples are evaluated, followed by one GP-guided acquisition, after which the gates are set to the best point found and the next window is visited. On subsequent window visits, the acquisition is guided by the existing Gaussian process model. The 19 + 1 split was found to perform best across a grid search from (19 + 1) to (2 + 18). On 4 dots, the scores after 100 measurements (mean ± standard error over 100 trials) were: 0.871 ± 0.005 (19 + 1), 0.860 ± 0.005 (10 + 10), 0.851 ± 0.005 (5 + 15) up to 0.814 ± 0.006 (2 + 18). The reason for this is that the objective is informative only close to the target and flat elsewhere. With few observations in a three-dimensional window, the GP has little structure to exploit, so broad coverage is worth more than early exploitation. Thanks to the reviewer, and to improve clarity, we have explained these hyperparameter choices in Appendix C in support of the fairness of our baselines.
We thank the reviewer for this suggestion. Further to point 1, we have also run a new baseline where our Kalman-guided virtualization framework is applied online for each of the classical optimizers. Here, we find the convergence behaviour to be similarly poor to when no virtualization is applied at all, concluding that classical algorithms, regardless of the extent to which they are granted access to virtualisation tools, are poorly suited to tuning larger quantum dot arrays. In line with our motivation for this work, we attribute this to the difficulty with which such algorithms navigate a non-convex and high-dimensional optimisation landscape. Failure to improve with virtualization can be attributed to virtualization amounting to a linear change of coordinates, which methods such as Nelder–Mead and L-BFGS are largely insensitive to. We have also applied our virtualization framework to MAPPO (see response to Reviewer 89h4), to illustrate how, in MARL settings, action factorization is an efficient strategy to improve agent coordination (substantiated theoretically by Appendix A). To reiterate our claims: virtualization removes the coupling between agents’ actions, rather than compensating for it as other MARL methods do, and RL outperforms classical optimisers because it learns from many devices how to translate measurements into informed actions, rather than searching each new device from scratch.
We value the points raised by the reviewer and believe our revision, thanks to them, has resulted in an overall stronger manuscript. We hope we have addressed the reviewer's points adequately, encourage any further discussion during the rebuttal period, and welcome them to increase their score.