acceptodds
Under review as a conference paper at ICLR 2027

Action-Factored Multi-Agent Reinforcement Learning for Scalable Quantum Device Tuning

Abstract

Cooperative multi-agent reinforcement learning is well suited to problems with large parameter spaces and exploitable local structure, however, if parameter coupling is strong, a non-stationary environment from the perspective of any individual agent can destabilize learning. We propose using a factored representation of the action space, learned online, to decouple agents. We demonstrate this approach on the tuning of electrostatically defined quantum dot arrays, a leading platform for quantum computing but whose calibration is hindered by cross-talk between gate electrodes. Our framework, **QADAPT**, optimizes voltages using local rewards obtained from individual agents. Through a parameter-sharing strategy, QADAPT also generalizes zero-shot to unseen array sizes, maintaining an approximately constant number of convergence steps. This modular approach provides a scalable route toward rapid calibration of large-scale quantum processors.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
quantum devices, marl, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_S3DiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the semiconductor quantum dot arrays optimization problem with multi-agent reinforcement learning. The authors propose using a factored representation of the action space, learned online, to decouple agents. The authore claim that the approach provides a scalable route toward rapid calibration of large-scale quantum processors.

Critical strengths
  1. The reviewer find the topic is well motivated. It is important to find a scalable way to calibrate quantum processors efficiently.
  2. The authors provide a good collection of ablation studies, including Kalman filter, various CNNs, IPPO, MAPPO, MADDPG, FACMAC...
  3. The authors demonstrated at least limited zero-shot scalability.
Critical weaknesses
  1. I think the authors should explain the requirements of using RL. If cross-capacitance matrix is got, then what is the reason of using a RL such as PPO?
  2. The reward and evaluations metric highly depend on the simulation software.
  3. There are some questions about the baseline fairness. (Detailed in the Questions Part.)
Questions for authors
  1. The authors hould clarify how to actually get those reward from actual devices if someone want to actually implement the algorithms.
  2. In Appendix C (Table 7), Bayesian optimization uses 19 Latin hypercube samples followed by only one GP-guided acquisition per window visit. Could the authors justify this configuration and evaluate whether allocating more evaluations to GP-guided optimization improves the baseline performance?
  3. Could the authors compare QADAPT against classical optimization or MARL baselines operating in the same virtualized action space? This would help distinguish the benefits of physics-informed preconditioning from those of the proposed multi-agent learning architecture.
1 comment
AuthorsOpenReview ↗

RL benefits discussed, rewards explained, baseline configurations justified

  1. We thank the reviewer for the opportunity to clarify our work. First, the cross-capacitance matrix is not available before tuning. It can only be estimated from CSDs that display charge transitions, which requires the device to already be partially tuned, and is why QADAPT estimates it online. Moreover, while an exact cross-capacitance matrix decouples the gates, it does not reveal where the target configuration lies. To illustrate the necessity of using RL, we have run an additional baseline in which the ground-truth cross-capacitance matrix is hypothetically known a priori, and applied from the start of each classical optimization round. The results are provided in Appendix D, and retain poor performance (see point 3). In our RL strategy, training across many simulated devices teaches QADAPT how the features in a CSD relate to the direction and distance to the target. Conversely, non-RL strategies reduce each measurement to a single scalar objective and must build their picture of the tuning landscape from scratch. In noise-dominated regions, such a scalar signal can be flat and give no indication of direction.

  2. Second, the reward is used only during training. At deployment, QADAPT acts solely on measured CSDs and gate voltages, so no reward, and hence no simulator, is required on hardware. Since the reward depends only on the distance to a physically defined target configuration (now specified in a revised Appendix E), any simulator or digital twin that captures the same physics would yield the same reward. In Appendix I, we apply the same framework to superconducting-qubit calibration, where the simulator and reward is different simulator, showing that our action-factorization strategy extends beyond a single problem setting. Our contribution therefore does not hinge on a particular reward structure. The reward is a task-specific component that can be exchanged, whereas QADAPT’s online action-space factorization and role-shared decentralized learning remain unchanged across both settings.

  3. BO is applied to sliding windows comprising two neighbouring plungers and their shared barrier (three voltages). For an array of N dots, this equates to N-1 windows. For the first visit to each window, 19 Latin hypercube samples are evaluated, followed by one GP-guided acquisition, after which the gates are set to the best point found and the next window is visited. On subsequent window visits, the acquisition is guided by the existing Gaussian process model. The 19 + 1 split was found to perform best across a grid search from (19 + 1) to (2 + 18). On 4 dots, the scores after 100 measurements (mean ±\pm standard error over 100 trials) were: 0.871 ±\pm 0.005 (19 + 1), 0.860 ±\pm 0.005 (10 + 10), 0.851 ±\pm 0.005 (5 + 15) up to 0.814 ±\pm 0.006 (2 + 18). The reason for this is that the objective is informative only close to the target and flat elsewhere. With few observations in a three-dimensional window, the GP has little structure to exploit, so broad coverage is worth more than early exploitation. Thanks to the reviewer, and to improve clarity, we have explained these hyperparameter choices in Appendix C in support of the fairness of our baselines.

  4. We thank the reviewer for this suggestion. Further to point 1, we have also run a new baseline where our Kalman-guided virtualization framework is applied online for each of the classical optimizers. Here, we find the convergence behaviour to be similarly poor to when no virtualization is applied at all, concluding that classical algorithms, regardless of the extent to which they are granted access to virtualisation tools, are poorly suited to tuning larger quantum dot arrays. In line with our motivation for this work, we attribute this to the difficulty with which such algorithms navigate a non-convex and high-dimensional optimisation landscape. Failure to improve with virtualization can be attributed to virtualization amounting to a linear change of coordinates, which methods such as Nelder–Mead and L-BFGS are largely insensitive to. We have also applied our virtualization framework to MAPPO (see response to Reviewer 89h4), to illustrate how, in MARL settings, action factorization is an efficient strategy to improve agent coordination (substantiated theoretically by Appendix A). To reiterate our claims: virtualization removes the coupling between agents’ actions, rather than compensating for it as other MARL methods do, and RL outperforms classical optimisers because it learns from many devices how to translate measurements into informed actions, rather than searching each new device from scratch.

We value the points raised by the reviewer and believe our revision, thanks to them, has resulted in an overall stronger manuscript. We hope we have addressed the reviewer's points adequately, encourage any further discussion during the rebuttal period, and welcome them to increase their score.

Reviewer_zfdSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes QADAPT (Quantum Arrays of Dots in Automated Parallel Tuning) for scalable control of quantum dot arrays. The main idea is to factorize the action space of the policy and use local policy predictions in a decentralized fashion to learn control parameters for centrally shared-parameters of the policy and value functions using the PPO algorithm. Results demonstrated improved fidelities across arrays of sizes 2 to 8 and across device architectures.

Critical strengths
  1. Well-written with sensibly constructed experiments and ablations.

  2. Clear improvements over the single agent PPO experiments in the ablation highlights the superiority of this specialized RL approach to their problem.

  3. Parameter sharing enables zero-shot transfer albeit this isn't shown rigorously enough. Is this transfer completely training free? Is this also across quantum device architectures?

Critical weaknesses
  1. Limited novelty for an ML conference venue is my main concern. Most of the paper's contribution is applying standard RL techniques to a quantum computing domain which is directly applicable for a physics journal but I fail to see any interesting takeaways that are general enough for domains outside quantum.

  2. I also find the baselines unconvincing. There should be various other distributed optimization methods e.g Distributed BO Self et. al. 2021 that could have been better fits here. A lot of the other baselines like BFGS and Nelder Mead could have been tested with the factorized framework.

  3. Note that Nature CNN beats or is very close to the proposed method for convergence.

Questions for authors

See above

1 comment
AuthorsOpenReview ↗

Novelty explained, scope clarified, baselines expanded

  1. We thank the reviewer, but would argue that our contributions go beyond a domain-specific application as we propose a general coordination mechanism for cooperative MARL. When agents act through coupled actions, each faces a non-stationary environment. Existing methods address this on the value side, through centralised critics, value factorisation or communication. QADAPT instead learns, online, a reparameterisation of the joint action space under which agent actions become approximately separable, preconditioning the problem with a provably bounded residual coupling (Appendix A). Table 1 shows that value-side coordination fails to match this under strong coupling (MAPPO 78/90/95%, FACMAC 0/0/1%, compared with 93/95/95% for QADAPT), and that action factorisation also improves a single-agent PPO learner (0/1/6% →\rightarrow 2/18/71%). In the revised manuscript, we also include a new baseline in which MAPPO operates in the action space produced by our Kalman-guided virtualization. Its convergence rises from 78/90/95% to 92/94/97%, matching QADAPT within uncertainty (Table 1). This isolates the effect of action-space factorization from the choice of critic, making action-space factorization not specific to our architecture: it is a coordination mechanism that is complementary to, and can be combined with, existing value-based MARL methods. We believe this to be an very useful finding for the ML community. Our mechanism requires only that an unknown, locally linear response can be estimated from observations. Appendix I (previously appendix H) applies this to superconducting qubits, where input and output dimensions, the physics of the couplings, and the observation modality all differ. In this context, good performance and generalisability to various device sizes is also demonstrated (Figures 9 & 10, previously 8 & 9). The same structure arises in coupled control problems beyond quantum technologies, for instance, power-grid voltage regulation and multi-zone climate control, which we believe could benefit from our strategy. We have revised the contributions and discussion to state this general takeaway more explicitly.

  2. We thank the reviewer for the suggestion. We do not claim that QADAPT is the only method suited to this problem; rather, we show that it outperforms well-established baselines from both communities it draws on (gradient-free optimisers widely used for quantum-dot tuning, including Gaussian-process BO, Nelder–Mead, L-BFGS and random search (Fig. 3), as well as model-based and multi-agent RL methods (DreamerV3 in Fig. 3; IPPO, MAPPO, MADDPG and FACMAC in Table 1). The distributed BO approach of Self et al. (2021) addresses a different problem class. There, each optimiser searches the full parameter space of a separate but related problem, and information sharing is possible because a single measurement yields the cost of every problem in the family. In quantum-dot tuning, agents instead control disjoint gates of a single cross-coupled device, and a measurement at one voltage configuration provides no cost evaluations elsewhere, so this mechanism does not carry over. Moreover, BO optimises each device from scratch with a surrogate that scales cubically in the number of observations; consistent with this, our GP-BO baseline slows sharply beyond 4 dots (Fig. 3), whereas QADAPT is trained once and generalizes across array sizes.

  3. We note that the Nature CNN variant is not a competing method but an ablation of QADAPT itself. Only the convolutional encoder fCNN(⋅)f_\mathrm{CNN}(\cdot) is replaced, while the virtualization module (CNN capacitance estimator fvirt(⋅)f_\mathrm{virt}(\cdot) and Kalman filter), the role-shared actor and critic heads, the voltage projection p(⋅)p(\cdot), the reward, and the PPO training strategy are unchanged. Its strong performance therefore reflects the robustness of QADAPT to the choice of encoder. At the most stringent criterion (2%), which is most relevant for tuning, the default lightweight encoder performs on par or better (93% vs. 91%) while requiring 40% fewer training epochs (150 vs. 250), which motivated our choice.

  4. Regarding the reviewer’s concern over the rigour of our zero-shot transfer, we have evaluated QADAPT on an out-of-distribution set of device parameters (Appendix B). Convergence rates remain within uncertainty of the in-distribution results (91/96/99% vs. 93/95/95%), at the cost of a moderate increase in steps to convergence. Table 1 now also reports uncertainty intervals. Crucially, both this experiment and the scaling experiments across array sizes were carried out without any retraining.

We greatly appreciate the reviewer's feedback and hope that the scope and impact of our work has been clarified to a satisfactory level. We warmly welcome any further discussion points during the rebuttal period and would be grateful should the reviewer deem an increased score appropriate.

Reviewer_2AigOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents a multi-agent RL framework QADAPT for autotuning semiconductor quantum dot arrays. The tuning difficulty in these devices stems largely from changing one plunger gate voltage, which prolongates the electrochemical potential of neighboring dots, so naive per-gate control induces interference and makes the joint action space grow combinatorially with array size. Experiment follows simulation via QArray on 2/4/6/8-dot arrays against gradient-free baselines, and DreamerV3.

Critical strengths
  • The idea is about learning a physically grounded change of coordinates to make the joint control problem approximately separable is theoretically scaffolded and empirically validated.
  • The results about scalability are strong and the benchmarking is unusually thorough for this literature.
Critical weaknesses
  • The zero shot scalability is overstated relative to what is actually demonstrated, and the authors own framing contains an issue with its bandit reduction. Zero shot in the ML community typically means transfer to an out of distribution setting. Here, the policy trained on 4 dots is deployed to 2/6/8 dots in the same simulator with the same parameter ranges, the same CSD resolution, and the same virtualization model. Parameter sharing makes the function class independent of array size, which is why transfer works but this is closer to the learned local controller is size agnostic than to zero shot generalization. A stronger demonstration would hold out device configurations, vary noise statistics at test time, or test on arrays with non identical gates.

  • The Dec-POMDP formalism lacks the abstraction for what is actually implemented, and the author lacks explanation about the consequences. The observations are CSD images, which are functions of the full device state and are what actually carry information about the state in the Dec-POMDP is just the voltage vector, which is fully observed by the agents. So the POMDP aspect is doing no work — the partial observability is an artifact of each agent only seeing its own CSDs, but the CSDs themselves are generated from the global state.

Questions for authors
  • The scaling experiments use the same simulator and parameter ranges for all array sizes. Were any held out device configurations tested or evaluated with different noise/latching parameters?

  • Is the lever arm αi\alpha_i used in reward normalization known to the agent at deployment, or only used during training to define rewards?

  • Given γ\gamma = 0 and reward advantages, what precisely does the Dec-POMDP formulation contribute beyond justifying the CTDE architecture? How will it change if the problem were framed directly as cooperative contextual bandits with shared policies?

1 comment
AuthorsOpenReview ↗

Zero-shot transfer substantiated, lever arm clarified, mathematical formulation tightened

  1. We thank the reviewer for this careful point. Indeed, our results demonstrate generalization to unseen array sizes (beyond those seen in training), albeit for an in-distribution set of simulation parameters. In our revision, we have deployed QADAPT on an out-of-distribution parameter set, whose unique ranges are now provided in Table 2 of Appendix B. For these new runs, we find convergence rates of 91/96/99% and a mean number steps to convergence of 24.7/21.3/16.1. These values now appear in Table 1, supporting our original claims on zero-shot transfer. Furthermore, we wish to emphasize that gate-dot and dot-dot capacitances are precisely what corresponds experimentally to devices with non-identical gate geometries. For example, larger gates yield larger gate–dot capacitances, and more closely spaced gates yield larger cross-capacitances. Non-uniform gate shapes would result in non-uniform capacitances. All results in Fig. 2 and Table 1 are obtained from 100 tuning runs on devices seeded with parameters distinct from those in the training set and that vary all of these characteristics. We have made this relationship explicit in Appendix B.

  2. The lever arm is a hidden quantity to all agents during training and used only for reward shaping. At deployment, the lever arm does not enter in any way. This is what allows QADAPT to generalise well to unseen devices, where the conversion between voltage and energy (which controls dot occupations and their tunnel coupling) differs. This is important, because the lever arm is also a quantity that is unknown experimentally. For completeness, we have added our reward functions to Appendix E (previously Appendix D).

  3. We thank the reviewer for this careful reading. Our formulation reflects what is accessible in experiment: gate voltages are set by the experimenter and hence known, whereas device parameters ξ\xi (capacitances, lever arms, noise characteristics) vary between devices and can only be inferred from local, noisy CSDs, each a two-dimensional slice of voltage space. Our simulator samples a new ξ\xi at each reset (Appendix B), which determines the target v∗\mathbf{v}^* and the rewards but is withheld from the agents, as on real hardware. The partial observability is therefore a property of the measurement process rather than an artefact of the architecture. We have revised our mathematical formulation and transition dynamics to include ξ\xi in Section 3.1.

    With γ=0\gamma = 0, temporal credit assignment is removed, but the sequential structure is not, and this is what distinguishes the Dec-POMDP from a cooperative contextual bandit. A bandit assumes contexts independent of the learner’s actions, whereas in tuning each action sets the voltages of the next measurement, so the policy determines which states it encounters. This concentrates training in the informative region where CSDs show transition lines. Our strict contextual-bandit baseline tests this directly: trained with the same architecture, reward and objective, but on independently sampled states, it achieves 0/0/0% convergence (Table 1). We find that the policy defaults to the centre of the voltage window. Hence, while the discount factor is incidental to performance (γ=0.9\gamma = 0.9: 89/94/97%), the closed-loop, on-policy interaction specified by the Dec-POMDP is essential.

We hope the reviewer is satisfied with our response, and we believe their input has objectively improved the manuscript and it's framing as a whole. We would greatly appreciate if the reviewer were to revise their scores, and we welcome any further comments during the rebuttal period.

Reviewer_89h4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, a physics‑informed multi‑agent RL framework of QADAPT is proposed for quantum‑dot‑array auto‑tuning, which uses CNN‑Kalman online crosstalk estimation to build a decoupled virtual action space. The proposed QADAPT enables zero‑shot generalization to larger device sizes..

Critical strengths

(1) The action transformation addresses a meaningful physical source of agent interference and provides a plausible explanation for improved coordination.

(2) The evaluation of the proposed method includes several classical and MARL baselines, component ablations, transfer across array sizes, and an additional superconducting-qubit setting.

Critical weaknesses

(1) All evaluations are conducted purely in simulation and the sim‑to‑real gap remains unvalidated.

(2) Real‑hardware artifacts such as parameter drift and hysteresis can degrade online cross‑capacitance estimation.

(3) Fabrication imperfections may blur the physical roles of different gates and hurt parameter‑sharing performance, and the framework is not validated for isolated‑regime quantum‑dot devices.

Questions for authors

(1) How do strong classical controllers and MAPPO perform when given the same online virtualization module?

(2) Can the main comparisons be repeated across independent training seeds, with explicit training budgets and uncertainty intervals?

(3) QADAPT assumes stationary cross‑couplings for its Kalman‑filter virtualization. How will it handle time‑varying drift present on real quantum‑dot devices?

1 comment
AuthorsOpenReview ↗

Additional baselines run, virtualisation and device drift clarified, confidence intervals provided

  1. We thank the reviewer for this good suggestion, allowing us to isolate the contribution of the virtualization module to tuning performance independently of the backbone algorithm. We have re-run all the classical optimizers and MAPPO with online virtualization, which can now be found in Appendix D and Table 1, respectively. Please refer to our response to Reviewer S3Di for a discussion on the classical optimizers. Here, we highlight that a centralized critic, as in MAPPO, is an alternative RL approach to addressing the same coordination problem, highlighted in our Related Work section. A centralized critic only helps when an agent’s own observation isn't enough to predict its return, i.e. when other agents’ actions affect its reward or dynamics. That coupling is exactly what virtualization removes. Once dynamics are decoupled, and each agent’s value depends only on its own observation, extra inputs to a centralized critic carry no useful information, and both centralized and decentralized variants give the same expected gradient. Consequently, the theoretical best performance of MAPPO with virtualisation can match QADAPT, which is consistent with our new results (93/95/95% convergence for QADAPT, 78/90/95% for MAPPO, and 92/94/97% for MAPPO w/ virt). Alongside IPPO (which is identical to QADAPT when virtualization is absent), these results convey that action factorization can be generally profitable, independent of the underlying MARL algorithm. The upside to QADAPT is that it doesn't rely on a centralized critic architecture, which has to learn to ignore inputs whose size grows with the number of agents. This invites spurious correlations and increases the variance of the gradient estimate, as found by Lyu et al., AAMAS 2021 and de Witt et al., 2020. A concatenated-input critic also ties the model to a fixed number of agents, which forgoes the scalability and generalizability of QADAPT, as stated in our Discussion.

  2. Unfortunately we do not have the computational resources to do multi-seed runs at this time. We reiterate, however, that each baseline in Table 1 uses a statistic over 100 tuning runs. Thanks to the reviewer, uncertainties have been added alongside these reported values, and those of additional baselines. Within these intervals, QADAPT outperforms all baselines without action factorization at the most stringent 2% radius, and its spread in steps to convergence is the lowest among them. (We note that a typo was spotted for the single agent PPO with virtualisation baseline and the 50.8 entry has been corrected to 15.8.)

  3. We distinguish two effects: (i) drift at a fixed voltage configuration, and (ii) changes in cross-capacitance as gate voltages are varied.

    (i) Charge noise in quantum-dot devices follows an approximately 1/f spectrum, with modern heterostructures reaching SE≈0.3\sqrt{S_E} \approx 0.3–1 μeV/Hz1~\mu\mathrm{eV}/\sqrt{\mathrm{Hz}} at 1 Hz (Lodari 2021; Paquelet Wuetz 2023; Stehouwer 2024). Such noise predominantly offsets charge transitions, rather than changing cross-capacitances. As shown in Fig. 2, QADAPT requires fewer than 20 steps for arrays of up to 8 dots. With experimental measurements on the order ∼1–10 s per step, a full tune-up would take a few minutes. Integrating a 1/f spectrum over this window gives RMS energy fluctuations of only a few µeV -- roughly three orders of magnitude below typical meV-scales that would change a dot's occupation. The single-step objective is robust to these small effects, as stated in Section 5.3

    (ii) The Kalman filter does not assume stationary cross-couplings: its state evolves as a random walk whose process noise explicitly models time variation. Updates are scaled by this process noise and inversely by the measurement noise. This lets the filter track cross-capacitances as the confinement potential changes in real devices, and would equally absorb any slow drift in the cross-couplings themselves (Sec. 3.2, Appendices A and H).

We hope our responses and the additional experiments and information have addressed the reviewer's concerns. If so, we would be grateful should the reviewer consider updating their assessment. We are happy to clarify anything further before the discussion period closes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised Manuscript Available

We thank all reviewers for their thorough assessment of our work. We have uploaded a revised manuscript incorporating all feedback, with changes highlighted in blue, so that reviewers can see the paper as close as possible to its final form. We warmly welcome further discussion of any points the reviewers feel have not yet been fully addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.