- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an LLM-based AI Scientist framework for automatically discovering hardware-constrained quantum error-correcting codes. The framework integrates LLM-generated executable code constructions with deterministic verification and iterative feedback.
The authors evaluate the framework using GPT, DeepSeek, and Claude on seven tasks based on Google's 105-qubit Willow topology, including four calibration-informed defect scenarios and three connectivity-design tasks. On defective layouts, the discovered subsystem codes achieve higher code and circuit fault distances than native surface-code baselines, while reducing CZ gate count and depth relative to distance-matched routed implementations. Additional experiments demonstrate improvements in logical-qubit capacity, code distance, and connectivity requirements under specified hardware constraints.
Critical strengths
- Verification-grounded LLM framework for hardware-aware code discovery. For the automated discovery, one of the crucial thing is to verify and evaluate results. The paper has done a careful analysis on this aspect.
Critical weaknesses
-
Restricted search space and limited generality. Although the framework is presented as an automated approach to subsystem-code discovery, it is restricted to CSS codes with separate X- and Z-type gauge generators, without considering general non-CSS subsystem codes. Moreover, several of the strongest reported constructions are based on known Bacon–Shor code families or stabilizer codes with .
-
Lack of ablation studies to validate the proposed self-evolving framework. A central contribution of the paper is the iterative discovery process driven by verification feedback and accumulated search memory. However, the paper does not provide controlled ablation studies to isolate the contributions of these components. In particular, comparisons with LLM search without feedback or memory, as well as conventional evolutionary or random search under matched computational budgets, are missing. Although the reported search trajectories show improvements over successive rounds, they do not establish whether these gains arise from the proposed feedback mechanism or simply from evaluating more candidates. This makes it difficult to assess the necessity and effectiveness of the key methodological contributions.
Questions for authors
See the weakness part: could the authors elaborate more on these two points?
7 comments
W2: classical search baselines and verification-feedback ablation
We thank the reviewer for this important suggestion. We agree that search trajectories alone are insufficient to establish the effectiveness of the proposed self-evolving mechanism, as improvements may also arise from evaluating more candidates. To address this concern, we conducted two complementary sets of experiments: comparisons with conventional non-LLM search methods and ablations of verification-driven feedback.
Classical search baselines. We introduce three conventional search strategies: structured random search, logical-guided simulated annealing, and mutation-population search. All three methods operate under the same hardware constraints and use the same deterministic code verifier as our framework. A short CP-SAT initialization provides generators satisfying the initial algebraic constraints. Each algorithm is evaluated with three independent random seeds, with approximately 12 CPU-hours shared across the nine runs per setting. The elapsed search time is comparable to that of our LLM searches. Across all seven settings, these classical approaches collectively generated 6,563,593 candidate proposals.
| Setting | GPT: full | DeepSeek: full | Best classical |
|---|---|---|---|
| 2C | [[49,1,36,7]] | [[49,1,36,7]] | [[24,1,0,3]] |
| 4C | [[49,1,36,7]] | [[49,1,36,7]] | [[19,1,0,3]] |
| 9C | [[25,1,16,5]] | [[28,1,18,4]] | [[49,1,0,3]] |
| 2Q + 2C | [[36,1,25,6]] | [[36,1,25,6]] | [[39,1,0,3]] |
| Protection () | [[54,2,38,6]] | [[32,2,18,4]] | [[14,1,0,3]] |
| Capacity () | [[75,7,52,5]] | [[75,7,52,5]] | [[13,1,6,3]] |
| Connectivity () | [[49,1,12,9]], | [[63,1,4,9]], | [[23,1,15,3]] |
The classical baselines achieve a maximum code distance of only three across all four defective-chip settings, whereas our LLM-guided framework reaches distances of seven, seven, five, and six, respectively. Furthermore, the classical searches fail to meet the prescribed protection targets in the three custom-layout tasks. These results demonstrate the effectiveness of LLM-guided program search over the conventional strategies evaluated here. Unlike predefined mutation operators, LLM-generated construction programs can coordinate structural changes to gauge generators, physical-qubit roles, measurement assignments, and hardware connectivity, providing greater flexibility in exploring this coupled design space.
Verification-feedback ablation. To examine whether iterative refinement contributes beyond independent LLM-based generation, we compare the full framework against no-feedback sampling using GPT and DeepSeek under the same task settings. The full framework incorporates previously verified constructions, code properties, and failure diagnostics into subsequent generations, whereas the no-feedback variant generates candidates independently without access to previous search outcomes.
| Setting | GPT: full | GPT: no feedback | DeepSeek: full | DeepSeek: no feedback |
|---|---|---|---|---|
| 2C | [[49,1,36,7]] | [[49,1,36,7]] | [[49,1,36,7]] | [[49,1,36,7]] |
| 4C | [[49,1,36,7]] | [[49,1,36,7]] | [[49,1,36,7]] | [[49,1,36,7]] |
| 9C | [[25,1,16,5]] | [[16,1,9,4]] | [[28,1,18,4]] | [[9,1,4,3]] |
| 2Q + 2C | [[36,1,25,6]] | [[48,1,35,6]] | [[36,1,25,6]] | [[48,1,35,6]] |
| Protection () | [[54,2,38,6]] | [[42,2,28,5]] | [[32,2,18,4]] | [[26,2,0,3]] |
| Capacity () | [[75,7,52,5]] | [[75,3,48,5]] | [[75,7,52,5]] | [[50,2,32,5]] |
| Connectivity () | [[49,1,12,9]], | [[81,1,64,9]], | [[63,1,4,9]], | [[81,1,64,9]], |
The results show clear improvements on the more challenging tasks. For GPT, verification feedback increases the distance from four to five on 9C and from five to six on Protection, improves logical capacity from three to seven on Capacity at (d=5), and reduces the maximum connectivity degree from six to four on Connectivity at (d=9). DeepSeek exhibits similar improvements across these tasks. On 2Q+2C, both models achieve (d=6), while the full framework identifies constructions using fewer data qubits. For 2C and 4C, both approaches reach the native-connectivity distance bound of seven, leaving no room for improvement in the primary objective.
Role of the self-evolving mechanism. The additional experiments provide complementary evidence for the two central components of our framework. The classical baselines highlight the effectiveness of LLM-guided construction search, while the no-feedback ablation demonstrates the benefits of incorporating verified search outcomes into subsequent generations. Although the current ablation evaluates feedback and accumulated search history jointly rather than isolating each component, the results provide direct empirical support for the proposed iterative discovery mechanism. We appreciate the reviewer's constructive suggestions, which have helped us strengthen the experimental validation of our framework.
We thank the reviewer for the constructive comments on the generality of our framework. We appreciate the opportunity to clarify the scope of our search space and the interpretation of the resulting code constructions.
Generality of CSS subsystem codes. Our current framework focuses on CSS subsystem codes, where X- and Z-type gauge generators are not required to commute with each other. While this does not cover general non-CSS subsystem codes, it already provides a large and nontrivial design space. Rather than restricting candidates to predefined code families, our framework jointly explores gauge generators, physical-qubit assignments, measurement layouts, and hardware connectivity across up to 105 physical qubits. These variables are strongly coupled through algebraic validity, logical protection, and hardware feasibility. The CSS setting also offers practical advantages in syndrome extraction and decoding, making it a natural starting point for hardware-aware subsystem-code design. Extending the framework to general non-CSS subsystem codes through a full binary symplectic representation is an interesting direction for future research.
Known code families versus search generality. We agree that several strong constructions identified by our framework belong to well-established code families, including the Bacon–Shor codes in the four defective-chip experiments. This outcome is consistent with the physical characteristics of the considered hardware. On highly local superconducting architectures, Bacon–Shor codes offer geometrically structured constructions with low-weight gauge measurements, making them particularly suitable candidates. Importantly, these code families were not imposed as search restrictions, but emerged from a broader search over gauge structures and hardware realizations. The custom-layout experiments further demonstrate the versatility of our approach, producing different constructions such as ([[50,2,4,7]]) and ([[49,1,12,9]]) under distinct connectivity constraints. These results highlight the value of automated search in identifying suitable code constructions for different hardware objectives.
Inclusion of stabilizer codes. Stabilizer codes are naturally included in the subsystem formalism as the special case (r=0). We retain this possibility to allow the search to explore different gauge structures according to the specified objective. For example, Claude identifies a ([[70,10,0,5]]) stabilizer code in the Capacity task, protecting ten logical qubits at distance five using 100 physical qubits and maximum connectivity degree six. This result illustrates how the framework accommodates both subsystem and stabilizer constructions within a unified hardware-aware search process.