- Rating
- 4(+2)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes QuRepair, a diagnosis-driven LLM program repair framework for multiple quantum programming frameworks. The method integrates framework-aware static analysis, API evidence, quantum circuit representation, and quantum semantic rules, and uses structured diagnosis to guide a large language model in iteratively generating repaired programs. The main contributions of the paper include: constructing a controlled fault injection method across Qiskit, Cirq, PennyLane, and OpenQASM; proposing a unified quantum behavior evaluation protocol; and systematically comparing multiple LLM debugging baselines. Experimental results show that QuRepair achieves an overall repair rate of 62.83% on 152 tasks, significantly higher than the strongest baseline’s 47.0%, and reaches 72.4% on the Bugs4Q-Robust real-world bug set. It also maintains advantages across multiple base models, validating the effectiveness of its quantum-specific evidence and diagnostic pipeline.
Critical strengths
(1) The topic is practically relevant: Quantum programs often suffer from the problem of “running but producing wrong results,” which traditional repair methods struggle to handle. This paper specifically studies such quantum semantic errors, giving it strong cross-disciplinary research value.
(2) The system design is relatively complete: QuRepair integrates static analysis, API evidence, circuit textualization, quantum rules, structured diagnosis, and LLM-based repair, and also supports multi-round iteration. It is fairly complete and implementable from an engineering perspective.
(3) The cross-framework experiments are valuable: It covers Qiskit, Cirq, PennyLane, and OpenQASM, and tries to construct aligned tasks to reduce interference from framework differences, making it easier to compare repair performance.
(4) The evaluation criteria are stricter: It does not only check whether a program can compile or run, but also compares the measurement probability distributions of the repaired program and the reference program, using KL divergence to judge whether their quantum behavior is consistent. This better reflects the characteristics of quantum programs.
(5) The results and generalization validation are relatively sufficient: Under GPT-4o, the overall repair success rate reaches 62.83%, clearly higher than the strongest baseline’s 47.0%. It is also validated across multiple frameworks, real bugs, and different base models, showing that the effectiveness does not depend entirely on a single LLM.
Critical weaknesses
(1) Insufficient methodological innovation: QuRepair is more like an engineering system that stitches together static analysis, API checking, circuit representation, rule-based diagnosis, LLM generation, and iterative feedback, so its originality is somewhat lacking.
(2) The role of Jev is unclear: The paper does not clearly explain Jev’s architecture, training process, or advantages over ordinary LLM-based diagnosis. Since Jev is an external model, it is difficult to determine whether the performance gain comes from structured diagnosis, the extra model calls, or Jev itself.
(3) Lack of component-level ablations: The system contains too many modules, but they are not tested one by one. Therefore, it is impossible to know which component actually brings the improvement, or to prove that “quantum-specific diagnosis” is truly essential.
(4) Baseline comparison is unfair: QuRepair uses more model calls, tool analysis, multiple rounds of iteration, and extra information, while the baselines are not aligned in terms of budget or conditions. Thus, its advantage may simply come from “more resources,” not necessarily from a better method.
(5) The benchmark may encourage rule adaptation: The test errors are highly correlated with the predefined quantum rules, so the system may have been optimized only for known mutation patterns. This does not prove that it can fix unknown errors, combined errors, or real-world development bugs.
(6) Quantum semantic reasoning is not deep enough: Although the paper discusses errors such as phase, non-commuting operation order, and qubit ordering, it mainly relies on predefined rules, textualized circuits, and LLMs. It does not propose new quantum semantic analysis or error attribution algorithms.
Questions for authors
- Compared with existing work, what new model, training objective, or repair algorithm does QuRepair propose? Or is it merely engineering integration?
- What are Jev’s architecture and training process? Do the gains come from structured diagnosis, extra model calls, or the model itself?
- Can you provide component-level ablations? How large is the independent contribution of quantum-specific diagnosis?
- Are the baselines aligned in model calls, tokens, candidate numbers, and feedback? Could the advantage come from a higher budget?
- Do the test errors highly overlap with the predefined rules? Can it generalize to unknown, combined, and real-world bugs?
- Beyond predefined rules and LLM reasoning, is there any new quantum semantic analysis or error attribution algorithm?
We thank the reviewer for recognizing the relevance of quantum program repair and our cross-framework evaluation. We address the concerns below.
W1/Q1 and W6/Q6: Novelty and Quantum-Semantic Analysis
QuRepair does not introduce a new pretrained model or training objective. Its contributions are (i) framework-aware, source-grounded quantum-semantic evidence for diagnosis-guided repair and (ii) validated, logically aligned fault injection across four frameworks.
QuRepair normalizes operations from Qiskit, Cirq, PennyLane, and OpenQASM, retaining operation types, qubit roles, parameters, measurement/reset events, source locations, and ordering. Conservative analyses identify potential cancellation, measurement-order anomalies, parameter/qubit-role inconsistencies, and missing operations in repeated blocks. A separate layer supplies quantum facts about noncommutativity, relative phase, qubit order, inverses, parameter provenance, and dynamic circuits. These signals are not automatically labeled faults; diagnosis uses the task specification, and evidence is updated after failed repairs.
For non-API faults, mutations originate in canonical Qiskit programs and are translated across frameworks. Retention requires translation fidelity, valid fault-family membership, and observable behavioral deviation; checks also cover modified locations and target structure. Operation-order mutations require explicit noncommutation checks. API faults use a separate process with 51 aligned logical groups validated in all four frameworks. The non-API corpus contains 541 validated groups (2,164 artifacts) spanning 34 benchmark families. This aligns logical fault effects, not textual edits. We claim neither a new equivalence algorithm nor a general semantic theory, and will clarify the relation to prior quantum testing work.
W2/Q2: Jev's Role
Jev is an externally trained typed decision model from TypeSafe AI; we did not develop or train it. It selects structured diagnoses from quantum-specific evidence, while GPT-4o generates repairs. We will specify its identifier, inputs, typed interface, and responsibilities; its architecture and training details are not fully disclosed.
On 608 instances, Jev diagnosis achieved 62.83% repair success versus 58.88% for GPT-4o diagnosis, with recorded costs of $37.57 versus \$83.94 (~55% less). Diagnostic inputs, GPT-4o repair model, feedback, and five-attempt limit were held fixed. However, implementation differences remain and token budgets were not equalized, so this is not a strictly fixed-budget causal comparison.
W3/Q3: Component Ablations
We ablated components on the same 608 instances, fixing the repair model, five-attempt limit, feedback, evaluator, and remaining components.
Quantum-semantic evidence has the largest effect, supporting its value beyond generic iterative repair. Other components contribute smaller individual gains. We will report these results in the revision.
W4/Q4: Baseline Fairness and Resources
We agree that baseline comparisons evaluate complete systems, not equalized inference budgets. Nevertheless, QuRepair does not uniformly use more resources. QBugLM, the strongest overall baseline, also uses GPT-4o repair and allows five attempts. On 608 instances:
QuRepair's counts include Jev. Its 15.83-point success advantage over QBugLM is thus not attributable to higher recorded model usage. Some baselines (e.g., RING) use fewer resources. We will distinguish these system-level comparisons from component ablations, which more directly isolate quantum-specific evidence under a fixed repair procedure.
W5/Q5: Taxonomy Overlap and Generalization
We acknowledge overlap between the mutation taxonomy and diagnostic rules. However, QuRepair receives no mutation labels, edited source locations, clean reference code, or reference distributions. Evidence comes solely from the current candidate and is recomputed after failures; detecting a quantum construct does not itself imply an error.
We also evaluated on Bugs4Q-Robust, a set of independently sourced, naturally occurring Qiskit defects. QuRepair passed official task-specific tests on 21/29 cases (72.4%). This supports transfer beyond our synthetic mutation process, but neither single-fault benchmarks nor this real-bug set establish generalization to arbitrary unseen fault categories or compound defects. We will state these limits explicitly.
We hope these results and clarifications address the reviewer's concerns.