acceptodds
Under review as a conference paper at ICLR 2027

CircuitPilot: Learning Circuit Optimization through Imitation and Self-Play

Abstract

Optimizing a circuit means searching a large space of equivalent circuits for a cheaper one, and a bounded optimizer must choose which rewriting and search procedures to run. We propose CircuitPilot, which learns these choices in two stages: a policy first imitates the optimal first actions found by exhaustive short-horizon search, then improves through single-player self-play on targets from policy-guided tree search. We apply one formulation to polynomial, Boolean, and quantum circuits, with a separate policy per domain; every returned circuit is checked for equivalence to its input. On 161 three-qubit development circuits, the quantum policy matches or improves on the cost of a step-limited Quasar configuration on 152 (70 strictly better). An imitation-only polynomial policy obtains lower total cost than each of three fixed schedules, though not lower than their per-input best, and 98 of 100 IWLS 2026 Boolean benchmarks can be encoded as model inputs.

Primary area:
reinforcement learning
Keywords:
Program synthesis, circuit optimization, reinforcement learning, imitation learning, self-play, equality saturation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hgYTOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces CircuitPilot, which learns to choose symbolic procedures for circuit optimization. The policy first imitates actions found by short symbolic searches, followed by self-play in the quantum setting. The framework includes polynomial, Boolean, and quantum environments with circuit-equivalence checks. The experiments evaluate quantum and polynomial optimization, while the Boolean study tests input encoding.

Critical strengths
  1. Learning which optimization procedures to apply and in what order is a reasonable and potentially useful idea.

  2. Circuit equivalence is checked separately from the learned policy, helping ensure that optimization preserves the original computation.

  3. The paper is clearly written and openly discusses the limitations of its results.

Critical weaknesses
  1. The main learning contribution is not sufficiently established. The central idea is to learn better optimization schedules, but the experiments do not convincingly show what this learning adds to the underlying symbolic optimizer. In the quantum experiments, the contribution of the learned controller is not isolated, and there is no comparison showing whether self-play improves over imitation alone. The results therefore provide limited evidence of a meaningful advance in learned circuit optimization.

  2. The reported improvements provide limited support for the approach. CircuitPilot is allowed 1,800 seconds per quantum circuit, while the tested Quasar configuration is limited to 60 seconds. For polynomial circuits, the gain over the best individual fixed schedule is only 0.25%, and the timed learned policies are slower. The tested portfolio of fixed schedules also achieves lower circuit cost than the portfolio containing a learned policy. These results leave the practical benefit of learned scheduling unclear.

  3. The evaluation remains preliminary. Both optimization studies use development data, and the quantum results were used to revise the rewrite rules. It is therefore unclear whether the improvements would hold on untouched test circuits. The Boolean study evaluates no optimization policy, so it does not provide further evidence for the learning approach.

Questions for authors
  1. How do fixed schedules, imitation alone, and imitation followed by self-play compare within the same symbolic environment?

  2. Do the gains hold on untouched test circuits under comparable optimization budgets?

1 comment
AuthorsOpenReview ↗

Author Response to Reviewer hgYT

We thank the reviewer for the careful evaluation and for recognizing the value of learning optimization procedures, independently verified circuit equivalence, and our discussion of limitations.

We particularly appreciate your emphasis on distinguishing the benefit of learning from that of the underlying symbolic optimizer. We agree that the original submission did not provide a sufficiently controlled comparison, and we have conducted new experiments specifically to address this issue.

Q1. What does self-play contribute beyond imitation and fixed optimization procedures?

Our new controlled ablation provides strong evidence that the second training stage produces substantial improvements over the original finite-horizon imitation procedure.

On 177 new development circuits, three imitation-only models obtain 670–695 CX. Continued imitation from a common checkpoint obtains 693 CX after 64 additional updates. In contrast, self-play from that checkpoint reaches 616 CX, reducing CX by 77 (95% CI [51, 106] saved) and total gates by 1,035 (27.6%). There are 32 CX improvements and no regressions.

The natural fixed schedule corresponding to the teacher's preferred sequence-resynthesis action obtains 655 CX, while self-play achieves 616. Together with the matched continued-imitation control, this comparison shows that the benefit of the second training stage is not explained merely by additional training updates or by following the teacher's preferred fixed procedure.

Thus, the controlled result establishes a substantial contribution of self-play beyond the original imitation teacher.

Q2. Do the improvements generalize, and are the computational comparisons fair?

We evaluated the final rewrite rules on 160 untouched circuits using a frozen protocol. They reduce total gate counts across three frozen schedules, with reductions of 55–66 gates and paired confidence intervals entirely below zero. This demonstrates transfer of the symbolic rule improvements beyond the development set.

We have also measured actual optimization times rather than relying on timeout caps. The 1,800-second limit reported originally is a maximum; actual deployment takes approximately 27 seconds per circuit in our controlled timing study. We additionally evaluate multiple Quasar configurations under explicit time budgets and report the resource conditions.

Under matched per-circuit wall-clock limits, CircuitPilot obtains 620 versus 737 CX against Quasar's default escalation configuration at 60 seconds and 616 versus 667 CX at 120 seconds on the new development panel. These results demonstrate competitive performance under constrained optimization budgets.

All reported outputs in the new 177-circuit learning ablation and 160-circuit rule-transfer evaluation passed exact circuit-equivalence checks.

These experiments clarify which gains arise from learning, which arise from symbolic optimization, and how the performance depends on resource budgets. We retain the polynomial result as a modest improvement over individual fixed schedules rather than a practical speed advantage, and explicitly distinguish Boolean encoding feasibility from learned optimization.

We believe the controlled learning ablation provides important evidence that was missing from the original submission, and we appreciate the reviewer's questions for motivating this more rigorous evaluation.

Reviewer_gZiNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CircuitPilot, a learning-based framework for circuit optimization that learns which optimization procedures to apply instead of following a fixed sequence of steps. The main contribution is a unified framework for selecting optimization procedures across polynomial, Boolean, and quantum circuits, using separate policies for each domain. The method combines imitation learning, where the policy learns from optimal actions found through short exhaustive searches, with self-play using policy-guided tree search to learn better optimization decisions. Another useful aspect is that the framework checks circuit equivalence to ensure that the optimized circuits preserve the original computation. The authors evaluate CircuitPilot on three circuit domains and showed some improvements for various domains. Overall, the paper provides an interesting approach to learning optimization, However, some contributions and practical benefits are not discussed properly.

Critical strengths

None

Critical weaknesses
  1. Lack of evidence for the benefit of self-play. One of the main contributions of the paper is combining imitation learning with self-play to improve circuit optimization. However, the experiments do not include a comparison between an imitation-only policy and a policy trained with both imitation and self-play. This makes it difficult to understand how much self-play actually contributes to the reported improvements. Since self-play is a central part of the proposed method, I think this comparison is important to justify the additional training complexity and demonstrate its effectiveness.
  2. Limited experimental evaluation and unclear practical benefits. Although the quantum results are promising, the evaluation is limited to 161 three-qubit development circuits, and the comparison with Quasar uses different time limits (1,800 seconds versus 60 seconds). Moreover, the rewrite rules were adjusted based on the development results, which lead to questions about generalization to unseen circuits. The paper also provides limited analysis of why the proposed quantum policy performs better on some circuits but worse on others. A more detailed analysis of these cases would help explain when and why the learned policy is effective. For polynomial circuits, the improvement over the best individual fixed schedule is very small (0.25%), while the learned policy requires more CPU time in the reported timing comparison. The Boolean experiments only evaluate whether circuits can be encoded, without showing optimization improvements. Overall, these limitations make it difficult to judge how well the approach generalizes and whether it offers a meaningful practical advantage over existing methods.
Questions for authors

Q1. How can we tell whether the policy learns meaningful, circuit-specific optimization decisions rather than simply learning which procedures tend to work best? Could the authors compare it with simpler heuristics or a policy that does not use the circuit's graph representation? Q2. Can the authors compare the final quantum policy with an imitation-only policy using the same rewrite rules? This would help clarify how much self-play contributes to the results. Q3. Since the final rewrite rules were adjusted based on the 161 development circuits, can the authors evaluate the final method on a separate, untouched test set to show how well it generalizes? Q4. How would CircuitPilot compare against a stronger Quasar configuration, with comparable computational budgets?

1 comment
AuthorsOpenReview ↗

Author Response to Reviewer gZiN

We sincerely thank the reviewer for the detailed and constructive questions. They motivated a substantially stronger experimental evaluation, including controlled learning ablations, analysis of policy behavior, untouched-circuit testing, and additional Quasar configurations.

Q1. Does the policy learn circuit-dependent decisions?

Our new analysis shows that the learned policy makes different procedure choices across circuit families, with different first-action patterns on commuting-layer and inverse-cancellation circuits. The improvement over imitation is also structurally concentrated: self-play saves 42 CX on commuting-layer circuits, 25 on inverse-cancellation circuits, and 10 on Clifford-word circuits. These findings characterize where self-play improves optimization and provide evidence of circuit-dependent policy behavior.

We also evaluated a strong fixed schedule that applies sequence resynthesis three times, corresponding to the procedure favored by the imitation teacher. This baseline obtains 655 CX and 3,253 total gates on 177 development circuits, whereas the self-play policy achieves 616 CX and 2,717 total gates.

Q2. How much does self-play contribute beyond imitation?

Starting from the same imitation checkpoint and controlling optimizer state, update counts, batches, rules, and evaluation conditions, self-play reduces aggregate CX from 693 to 616 (−77, paired 95% CI [−106, −51]) and total gates by 27.6% on 177 new development circuits. It improves 32 circuits in CX and worsens none. Continued imitation shows no comparable improvement, even with substantially more updates.

This directly isolates the contribution of the second training stage under the released rule set. We distinguish this controlled comparison from the separate evaluation required for the paper's final rule set.

Q3. Do the revised rewrite rules generalize to untouched circuits?

Yes, for the symbolic rule revision. We evaluated the final rules on 160 untouched circuits using a frozen protocol. Across three fixed schedules, total gates decrease by 55, 62, and 66 respectively; all paired 95% confidence intervals exclude zero. This shows that the rule improvements are not confined to the circuits used during development. We distinguish this finding from learned-policy generalization, which requires its own held-out evaluation.

Q4. What happens with stronger Quasar configurations and comparable budgets?

We evaluated multiple Quasar configurations under strict per-circuit time limits. An interesting result is that CircuitPilot compares favorably against Quasar's default escalation configuration at shorter budgets. On the 177-circuit development panel, at 60 seconds the totals are 620 CX for CircuitPilot versus 737 for Quasar's defaults; at 120 seconds they are 616 versus 667. At 300s, the results become statistically indistinguishable (616 vs. 621 CX). On the untouched panel, the fixed CircuitPilot schedule also compares favorably against this configuration at 30–120 seconds.

We appreciate the reviewer's emphasis on learning-specific evidence and generalization. The additional experiments allow us to provide substantially more precise answers to both questions and to delimit the practical contribution more carefully.

Reviewer_onsWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies circuit optimization: searching through the space of equivalent circuits to find the one with the lowest cost. The paper proposes a two stage method of optimizing a circuit using imitation learning followed by reinforcement learning. Specifically, the policy imitates bounded length sequence searches found through exhaustive search which are followed by single player self play for further optimization. The framework can be applied in a variety of circuits, namely, polynomial, boolean and quantum circuits.

Critical strengths

The motivation of the paper is clear. Finding the cheapest circuit from a class of equivalent circuits is a hard search problem. Using exhaustive search for this problem can saturate time and memory resources.

Critical weaknesses

Novelty: The paper presents imitation learning followed by self-play, in the style of AlphaGo, as a contribution. This recipe has already been applied to circuit and logic optimization, for example in Pirlls: Pretraining with imitation and rl finetuning for logic synthesis, which the paper does not cite.

Experimental Evaluation: Results come from a single training run on development data, and no timing comparison is reported.

Missing related work: The paper omits a lot of relevant literature, e.g. Xu et al. Optimizing quantum circuits, fast and slow

Missing ablations: The authors do not illustrate how self-play contributes to the improvement of the policy

Questions for authors

How much self-play improves the policy? Some comparisons seem unfair: CircuitPilot has a 30x larger runtime for quantum circuit optimization compared to Quasar. Can you clarify how many resources are used?

1 comment
AuthorsOpenReview ↗

Author Response to Reviewer onsW

We thank the reviewer for the careful assessment and for recognizing the motivation and soundness of our framework. Your concern about the missing self-play ablation was particularly valuable: it motivated a controlled experiment that now provides direct evidence for the contribution of the second training stage.

Does self-play actually improve the policy? Yes, substantially over the original imitation-based training. Starting from the identical imitation checkpoint, with matched optimizer state, batches, update counts, rewrite rules, and evaluation protocol, self-play reduces aggregate CX from 693 to 616 on 177 new development circuits (−11.1%, paired 95% CI [−106, −51]). It improves 32 circuits and worsens none. Total gates fall from 3,752 to 2,717 (−27.6%), with corresponding improvements in depth. Continued imitation does not achieve these gains, even after 256 additional updates. Self-play also outperforms the best of three independently initialized imitation policies by 54 CX.

Where does the second stage help? The gains over imitation are concentrated in particular circuit families: self-play saves 42 CX on commuting-layer circuits, 25 on inverse-cancellation circuits, and 10 on Clifford-word circuits. The learned policy also makes different procedure choices across these families, which helps characterize the improvement from self-play

What about computational resources? We agree that the original 1,800-second limit could be misleading: it is a maximum cap, not typical execution time. Our same-node measurements show approximately 27 seconds per circuit for the self-play policy. We also clarify the training cost: 64 self-play updates require approximately 150 CPU-hours of symbolic search, whereas the matched imitation control equalizes training updates rather than total compute. These measurements provide a substantially more precise accounting of the resources used.

Novelty. We appreciate the reviewer's reference to PIRLLS and have incorporated it, together with GUOQ and other relevant work, into the revised manuscript. We agree that combining imitation learning with RL is not itself novel. Our contribution lies in how these stages interact with bounded symbolic optimization. Unlike PIRLLS, which imitates high-quality synthesis trajectories and performs PPO finetuning for individual target circuits, CircuitPilot uses exhaustive bounded search to identify the complete set of optimal first actions under specified search budgets, providing exact, tie-aware supervision rather than trajectory-level demonstrations. Self-play then extends beyond the teacher's limited search horizon through policy-guided planning, with improvements amortized across a distribution of circuits. Crucially, the resulting policy is reusable and requires neither per-circuit RL finetuning nor tree search at deployment. Our new controlled ablation further demonstrates the practical role of this distinction: self-play learns to invoke a useful optimization procedure that the finite-horizon teacher cannot label, whereas continued imitation fails to recover it.

We believe the new controlled experiments directly address the central missing evidence identified in your review and substantially clarify the technical contribution.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Additional Experiments and Manuscript Revisions

We sincerely thank the Area Chair and all three reviewers for their careful evaluations and constructive feedback. The reviews identified an important gap in our original submission: the contribution of self-play had not been sufficiently isolated from imitation learning and the underlying symbolic optimizer. We have conducted substantial additional experiments to directly address this concern, along with the questions about generalization, runtime fairness, and novelty.

1. Controlled evidence for self-play. On 177 new development circuits, we compared self-play against continued imitation from the identical checkpoint, controlling optimizer state, update counts, batches, rewrite rules, and evaluation conditions. Self-play reduces aggregate CX from 693 to 616 (−11.1%, paired 95% CI [−106, −51]) and total gates from 3,752 to 2,717 (−27.6%), improving 32 circuits in CX with no regressions. Continued imitation does not achieve comparable gains, even with additional training updates. Three independently initialized imitation-only models also obtain higher CX totals (670–695).

2. Understanding the contribution of self-play. Our additional analysis shows that self-play improves upon bounded-search imitation supervision and produces different optimization decisions across circuit families. The gains over imitation are also structure-dependent, with improvements concentrated in commuting-layer, inverse-cancellation, and Clifford-word circuits. We further evaluated two fixed-schedule controls: the self-play policy improves upon the teacher-preferred sequence-resynthesis schedule (616 vs. 655 CX). Together, these results establish a substantial benefit over the original imitation-based training and teacher-preferred baseline.

3. Untouched-circuit evaluation. We evaluated the final symbolic rewrite rules on 160 untouched circuits under a frozen protocol. Across three fixed schedules, total gates decrease by 55–66, with paired 95% confidence intervals excluding zero. This provides evidence that the rule improvements transfer beyond the development panel; we distinguish it from held-out generalization of the learned policy.

4. Fairer runtime and baseline comparisons. We clarified that the originally reported 1,800-second limit is a maximum cap, not typical deployment time; measured execution averages approximately 27 seconds per circuit. Under matched per-circuit wall-clock limits, CircuitPilot obtains 620 vs. 737 CX at 60 seconds and 616 vs. 667 CX at 120 seconds against Quasar's default step-escalation configuration. We also clarify that performance depends on the Quasar configuration and distinguish matched training updates from matched training compute.

5. Novelty, correctness, and scope. We expanded the related-work discussion to include PIRLLS and GUOQ, clarifying CircuitPilot's distinctive combination of exact, tie-aware finite-horizon supervision, cross-circuit self-play, and search-free policy deployment. All outputs in the new learning-ablation and rule-transfer evaluations passed exact circuit-equivalence checks. We also clarify the more limited claims supported by the polynomial and Boolean experiments.

We have incorporated the new evidence and clarifications into the revised manuscript. We believe the controlled self-play ablation, circuit-family analysis, and expanded baseline comparisons directly address the central experimental concerns raised across the reviews and substantially strengthen the paper's empirical contribution. We appreciate the reviewers' feedback and the opportunity to improve the work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.