acceptodds
Under review as a conference paper at ICLR 2027

QuRepair: Diagnosis-Guided LLM Repair for Multi-Framework Quantum Software

Abstract

Large language models (LLMs) have shown strong performance in classical program repair, but quantum programs introduce challenges arising from quantum-specific operations, semantics, and programming frameworks. We introduce QuRepair, a structured quantum-aware repair framework in which static linter diagnostics, API evidence, and quantum-semantic rules inform a structured typed diagnosis produced by a decision model (Jev). A separate repair LLM uses this diagnosis and a textual circuit rendering to iteratively repair the program. For controlled evaluation, we also introduce a multi-framework fault-injection methodology spanning Qiskit, Cirq, PennyLane, and OpenQASM. Faults are defined by their intended effect and aligned across frameworks through canonical-source mutation and validated translation, with API faults realized through framework-specific mutations. Real-world Bugs4Q-Robust faults provide external validation. Evaluating five general-purpose and quantum-specific baselines, we find that existing methods vary substantially across frameworks and struggle on several challenging quantum-semantic faults, including relative-phase and operation-order errors. With GPT-4o, QuRepair achieves 62.83% overall repair success, 15.83 percentage points above the strongest baseline at 47.0%, with the highest success on three frameworks and a tie with the strongest baseline on OpenQASM. It also leads on real-world Bugs4Q-Robust faults (72.4% vs. 62.1% for the next-best method), and its advantage over direct repair persists across four underlying LLMs. These results show that structured quantum-specific evidence improves LLM-based quantum program repair across frameworks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Automated Program Repair, Quantum Program Debugging, Quantum Software Engineering, Program Analysis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_k4vHOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes QuRepair, a diagnosis-driven LLM program repair framework for multiple quantum programming frameworks. The method integrates framework-aware static analysis, API evidence, quantum circuit representation, and quantum semantic rules, and uses structured diagnosis to guide a large language model in iteratively generating repaired programs. The main contributions of the paper include: constructing a controlled fault injection method across Qiskit, Cirq, PennyLane, and OpenQASM; proposing a unified quantum behavior evaluation protocol; and systematically comparing multiple LLM debugging baselines. Experimental results show that QuRepair achieves an overall repair rate of 62.83% on 152 tasks, significantly higher than the strongest baseline’s 47.0%, and reaches 72.4% on the Bugs4Q-Robust real-world bug set. It also maintains advantages across multiple base models, validating the effectiveness of its quantum-specific evidence and diagnostic pipeline.

Critical strengths

(1) The topic is practically relevant: Quantum programs often suffer from the problem of “running but producing wrong results,” which traditional repair methods struggle to handle. This paper specifically studies such quantum semantic errors, giving it strong cross-disciplinary research value.

(2) The system design is relatively complete: QuRepair integrates static analysis, API evidence, circuit textualization, quantum rules, structured diagnosis, and LLM-based repair, and also supports multi-round iteration. It is fairly complete and implementable from an engineering perspective.

(3) The cross-framework experiments are valuable: It covers Qiskit, Cirq, PennyLane, and OpenQASM, and tries to construct aligned tasks to reduce interference from framework differences, making it easier to compare repair performance.

(4) The evaluation criteria are stricter: It does not only check whether a program can compile or run, but also compares the measurement probability distributions of the repaired program and the reference program, using KL divergence to judge whether their quantum behavior is consistent. This better reflects the characteristics of quantum programs.

(5) The results and generalization validation are relatively sufficient: Under GPT-4o, the overall repair success rate reaches 62.83%, clearly higher than the strongest baseline’s 47.0%. It is also validated across multiple frameworks, real bugs, and different base models, showing that the effectiveness does not depend entirely on a single LLM.

Critical weaknesses

(1) Insufficient methodological innovation: QuRepair is more like an engineering system that stitches together static analysis, API checking, circuit representation, rule-based diagnosis, LLM generation, and iterative feedback, so its originality is somewhat lacking.

(2) The role of Jev is unclear: The paper does not clearly explain Jev’s architecture, training process, or advantages over ordinary LLM-based diagnosis. Since Jev is an external model, it is difficult to determine whether the performance gain comes from structured diagnosis, the extra model calls, or Jev itself.

(3) Lack of component-level ablations: The system contains too many modules, but they are not tested one by one. Therefore, it is impossible to know which component actually brings the improvement, or to prove that “quantum-specific diagnosis” is truly essential.

(4) Baseline comparison is unfair: QuRepair uses more model calls, tool analysis, multiple rounds of iteration, and extra information, while the baselines are not aligned in terms of budget or conditions. Thus, its advantage may simply come from “more resources,” not necessarily from a better method.

(5) The benchmark may encourage rule adaptation: The test errors are highly correlated with the predefined quantum rules, so the system may have been optimized only for known mutation patterns. This does not prove that it can fix unknown errors, combined errors, or real-world development bugs.

(6) Quantum semantic reasoning is not deep enough: Although the paper discusses errors such as phase, non-commuting operation order, and qubit ordering, it mainly relies on predefined rules, textualized circuits, and LLMs. It does not propose new quantum semantic analysis or error attribution algorithms.

Questions for authors
  1. Compared with existing work, what new model, training objective, or repair algorithm does QuRepair propose? Or is it merely engineering integration?
  2. What are Jev’s architecture and training process? Do the gains come from structured diagnosis, extra model calls, or the model itself?
  3. Can you provide component-level ablations? How large is the independent contribution of quantum-specific diagnosis?
  4. Are the baselines aligned in model calls, tokens, candidate numbers, and feedback? Could the advantage come from a higher budget?
  5. Do the test errors highly overlap with the predefined rules? Can it generalize to unknown, combined, and real-world bugs?
  6. Beyond predefined rules and LLM reasoning, is there any new quantum semantic analysis or error attribution algorithm?
1 comment

We thank the reviewer for recognizing the relevance of quantum program repair and our cross-framework evaluation. We address the concerns below.

W1/Q1 and W6/Q6: Novelty and Quantum-Semantic Analysis

QuRepair does not introduce a new pretrained model or training objective. Its contributions are (i) framework-aware, source-grounded quantum-semantic evidence for diagnosis-guided repair and (ii) validated, logically aligned fault injection across four frameworks.

QuRepair normalizes operations from Qiskit, Cirq, PennyLane, and OpenQASM, retaining operation types, qubit roles, parameters, measurement/reset events, source locations, and ordering. Conservative analyses identify potential cancellation, measurement-order anomalies, parameter/qubit-role inconsistencies, and missing operations in repeated blocks. A separate layer supplies quantum facts about noncommutativity, relative phase, qubit order, inverses, parameter provenance, and dynamic circuits. These signals are not automatically labeled faults; diagnosis uses the task specification, and evidence is updated after failed repairs.

For non-API faults, mutations originate in canonical Qiskit programs and are translated across frameworks. Retention requires translation fidelity, valid fault-family membership, and observable behavioral deviation; checks also cover modified locations and target structure. Operation-order mutations require explicit noncommutation checks. API faults use a separate process with 51 aligned logical groups validated in all four frameworks. The non-API corpus contains 541 validated groups (2,164 artifacts) spanning 34 benchmark families. This aligns logical fault effects, not textual edits. We claim neither a new equivalence algorithm nor a general semantic theory, and will clarify the relation to prior quantum testing work.

W2/Q2: Jev's Role

Jev is an externally trained typed decision model from TypeSafe AI; we did not develop or train it. It selects structured diagnoses from quantum-specific evidence, while GPT-4o generates repairs. We will specify its identifier, inputs, typed interface, and responsibilities; its architecture and training details are not fully disclosed.

On 608 instances, Jev diagnosis achieved 62.83% repair success versus 58.88% for GPT-4o diagnosis, with recorded costs of $37.57 versus \$83.94 (~55% less). Diagnostic inputs, GPT-4o repair model, feedback, and five-attempt limit were held fixed. However, implementation differences remain and token budgets were not equalized, so this is not a strictly fixed-budget causal comparison.

W3/Q3: Component Ablations

We ablated components on the same 608 instances, fixing the repair model, five-attempt limit, feedback, evaluator, and remaining components.

ConfigurationSuccessΔ (pp)
Full QuRepair62.83%—
No quantum-semantic layer55.43%−7.40
No API evidence62.17%−0.66
No circuit rendering61.51%−1.32
No linter62.01%−0.82

Quantum-semantic evidence has the largest effect, supporting its value beyond generic iterative repair. Other components contribute smaller individual gains. We will report these results in the revision.

W4/Q4: Baseline Fairness and Resources

We agree that baseline comparisons evaluate complete systems, not equalized inference budgets. Nevertheless, QuRepair does not uniformly use more resources. QBugLM, the strongest overall baseline, also uses GPT-4o repair and allows five attempts. On 608 instances:

MetricQBugLMQuRepair
Repair success47.0%62.83%
Model calls12,2673,745
Tokens71.36M26.29M
Inference cost$139.00$37.57

QuRepair's counts include Jev. Its 15.83-point success advantage over QBugLM is thus not attributable to higher recorded model usage. Some baselines (e.g., RING) use fewer resources. We will distinguish these system-level comparisons from component ablations, which more directly isolate quantum-specific evidence under a fixed repair procedure.

W5/Q5: Taxonomy Overlap and Generalization

We acknowledge overlap between the mutation taxonomy and diagnostic rules. However, QuRepair receives no mutation labels, edited source locations, clean reference code, or reference distributions. Evidence comes solely from the current candidate and is recomputed after failures; detecting a quantum construct does not itself imply an error.

We also evaluated on Bugs4Q-Robust, a set of independently sourced, naturally occurring Qiskit defects. QuRepair passed official task-specific tests on 21/29 cases (72.4%). This supports transfer beyond our synthetic mutation process, but neither single-fault benchmarks nor this real-bug set establish generalization to arbitrary unseen fault categories or compound defects. We will state these limits explicitly.

We hope these results and clarifications address the reviewer's concerns.

Reviewer_bz8AOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a diagnosis-guided framework, namely QuRepair, for LLM-based quantum program repair. It combines static analysis, API evidence, quantum-semantic rules, and a Jev-based diagnosis to guide iterative repair. The authors also introduce a controlled fault-injection methodology covering Qiskit, Cirq, PennyLane, and OpenQASM. Experiments show that QuRepair outperforms existing debugging methods on the controlled benchmark and achieves encouraging results on real-world quantum software bugs.

Critical strengths
  • The paper addresses a relevant problem in LLM-based quantum program repair.
  • The evaluation covers five baselines and four quantum programming frameworks.
  • The paper includes controlled fault injection and real-world bug evaluation.
Critical weaknesses
  • The contribution of the proposed structured diagnosis is not established. QuRepair combines several components, including framework-aware linters, API evidence, quantum-semantic rules, circuit rendering, and Jev-based diagnosis. However, the evaluation compares the complete pipeline against existing debugging methods without isolating these components. The direct-repair comparison also removes the entire pipeline, including its iterative feedback mechanism. As the authors acknowledged in Section 6, the individual contributions of the evidence channels and Jev diagnosis remain unexamined. Consequently, the experiments do not demonstrate whether structured diagnosis is responsible for the improvements. This is particularly important because diagnosis-guided repair is the central contribution of the paper.

  • The evaluation provides limited evidence of improvements on challenging quantum-semantic faults. A major motivation of QuRepair is the difficulty of repairing quantum-semantic errors that are not exposed by conventional execution feedback. However, the primary evaluation contains 105 local faults, 22 API faults, and only 25 quantum-semantic faults. Of these 25 semantic faults, 12 come from the seven-qubit Steane code family. Table 1 reports repair difficulty by fault type for the existing methods, but the corresponding QuRepair results are not presented. Table 2 instead aggregates QuRepair's performance by framework. As a result, it is unclear how much of the overall improvement comes from repairing quantum-semantic faults, particularly relative-phase and noncommuting operation-order errors.

  • The semantic correctness criterion may overestimate successful repairs. The controlled evaluation determines repair success using the KL divergence between the reference and candidate output distributions, with a threshold of 0.05. However, this threshold permits noticeable differences in observable behavior. For example, the distributions (0.5, 0.5) and (0.65, 0.35) have a KL divergence of approximately 0.047 and would satisfy the stated criterion despite a 15-percentage-point difference in output probabilities. The experiments also use a fixed simulation seed and 16,384 shots, without reporting sensitivity to the evaluation threshold. Thus, the reported repair success rates depend on a correctness criterion whose tolerance is not sufficiently justified.

  • The comparisons do not fully control differences in debugging procedures and model resources. The primary evaluation uses GPT-4o as the repair model for all methods, which provides a useful common basis. However, QuRepair additionally uses Jev as a separate decision model. The compared methods also differ in their generation procedures, feedback availability, and repair budgets. For example, RING uses a one-shot Top@3 evaluation, while QuRepair performs up to five feedback-guided repair iterations. QBugLM provides a closer comparison because it also uses iterative localization and repair, but its information sources and diagnosis procedure differ. These comparisons are useful at the system level, yet they do not clearly distinguish the effects of quantum-specific information from those of the overall repair procedure.

  • The real-world evaluation provides limited evidence of generalization. The external evaluation uses 29 reproducible Bugs4Q-Robust cases, all from Qiskit. QuRepair successfully repairs 21 cases, compared with 18 for the strongest baseline. While this result is encouraging, the improvement amounts to only three additional repairs on a relatively small dataset. The paper does not establish whether this difference is statistically reliable. Moreover, the external evaluation does not cover Cirq, PennyLane, or OpenQASM. The evidence therefore supports preliminary generalization to real Qiskit bugs, but not broader real-world effectiveness across quantum programming frameworks.

Questions for authors
  • What is the individual contribution of Jev-based diagnosis to QuRepair's repair success?
  • How does QuRepair perform on each quantum-semantic fault category?
  • How sensitive are the repair success rates to the KL-divergence threshold?
  • How does QuRepair compare with the strongest baselines under matched repair and inference budgets?
  • How statistically reliable is the three-case improvement on Bugs4Q-Robust?
3 comments

We thank the reviewer for the careful assessment; the new experiments below are in the revised PDF.

Short answers

  1. Q1 (Jev). Small for success; its measured benefit is mainly efficiency. Without any diagnosis stage, QuRepair repairs 12 fewer instances at higher cost and time; a GPT-4o diagnoser repairs 24 fewer at 2.2× the cost. Of the four component ablations, only the semantic-layer removal is significant.
  2. Q2 (semantic categories). Revised Table 1 now compares QuRepair with the strongest baseline in all eight semantic families: QuRepair repairs 59/100 semantic jobs against 22/100 for QBugLM, so 37 of its 96-job margin over QBugLM comes from these 100 jobs (Part 2).
  3. Q3 (KL threshold). Stable across cutoffs: from 0.01 to 0.10, QuRepair's success stays at 62.01–62.83% and its margin over QBugLM at 15.63–16.45 points; five fresh simulator seeds give 60.20–60.86% (Part 2).
  4. Q4 (matched budgets). We ran no budget-matched experiment, but QBugLM, the strongest baseline, has the same five-attempt limit and uses more calls, tokens, and dollars, yet repairs 96 fewer instances (Part 3).
  5. Q5 (Bugs4Q-Robust). Not reliable on its own: with a net difference of three tasks (21 vs. 18 of 29), an exact two-sided McNemar test cannot give p below 0.25. We present it as evidence of transfer to real Qiskit bugs, not of statistical superiority (Part 3).

W1/Q1: Contribution of Structured Diagnosis and Individual Components

Diagnosis stage (Q1). As the reviewer notes, direct repair also lacks iterations and feedback (Section 5.3). The new no-diagnosis run removes only the diagnosis stage, passing the evidence straight to GPT-4o with no Jev or other diagnoser. All runs below share the 608 instances, GPT-4o repair, five attempts, feedback, and evaluator.

ConfigurationRepaired / 608Recorded costTime / instance
Full QuRepair (Jev), as reported382 (62.83%)$37.5735.88 s
No diagnosis stage370 (60.86%)$41.5647.60 s
GPT-4o diagnosis instead of Jev358 (58.88%)$83.94110.9 s

The reported 382/608 successful repairs come from one complete evaluation across all four frameworks and serve as the control for the component-ablation experiments. Relative to this full configuration, removing the diagnosis stage yields 12 fewer repairs (1.97 percentage points). We do not report a paired significance test for this diagnosis-stage comparison because the no-diagnosis run's aligned job-level outcomes were not retained in the audited archive.

Component ablations (Jev kept; paired against the 382/608 run). Exact McNemar tests on discordant jobs; paired Wald 95% CIs.

Removed component (Jev kept)Repaired / 608Full-only / ablated-only jobsDrop vs. full run, pp [95% CI]Exact McNemar p
None (four-framework run)382———
Quantum-semantic layer33774 / 297.40 [4.18, 10.62]< 0.001
API evidence37838 / 340.66 [−2.08, 3.39]0.724
Circuit rendering37437 / 291.32 [−1.30, 3.93]0.389
Linter37742 / 370.82 [−2.04, 3.69]0.653

Only the semantic-layer removal has a statistically distinguishable effect. It also changes the rules supplied to Jev, so it removes the semantic evidence pathway as a whole. The other three channels show small drops (0.66–1.32 percentage points), but their confidence intervals include zero, so we do not claim statistically significant contributions for them.

What this means for the central claim. The evidence supports a specific claim: quantum-semantic diagnostic evidence improves repair, and Jev's measured benefit is lower cost and time, but the advantage over existing methods does not depend on Jev. Both GPT-4o-only variants still beat QBugLM (370 and 358 vs. 286 of 608), at lower recorded cost (Part 3).

W2/Q2: Performance on Quantum-Semantic Faults

Revised Table 1 (Section 5.1) reports QuRepair and the strongest baseline for every semantic family, using one verdict per job from the job logs:

Semantic familyTasksQuRepairQBugLMBest other baseline
QFT-region corruption24/84/8LDB 7/8
Basis-conversion error27/82/8LDB 5/8
Reset removal13/42/4LDB 2/4
Phase-rotation error12/41/4LDB/RING/SD 1/4
Qubit-ordering error36/121/12LDB/RING 3/12
Measurement permutation12/41/4LDB 2/4
Relative-phase error1233/4811/48RING 1/48
Noncommuting operation order32/120/12RING 2/12
Semantic cohort2559/10022/100–

(Please note that SD = Self-Debugging). 

This answers how much of the overall gain comes from semantic faults: on these 100 jobs QuRepair repairs 37 more than QBugLM, out of its 96-job margin over QBugLM on all 608. Pooled over the five baselines, the cohort is repaired in only 69/500 jobs (13.8%), and relative-phase and noncommuting operation-order errors in 5.0% (12/240 and 3/60). Against the strongest baseline in each family, QuRepair leads in five, ties in two, and trails LDB on QFT-region corruption (4/8 vs. 7/8); its largest gain is on relative-phase errors (33/48 vs. 11/48).

QuRepair repairs semantic-cohort faults about as often as local ones (59.0% vs. 59.5%, 250/420). Grouped by mutation operator instead, semantic faults are somewhat harder: 74/132 jobs (56.1%) on the 33 semantic-operator tasks, eight of them in the local cohort, vs. 235/388 (60.6%) on the 97 local-operator tasks. For the 13 local and API categories, QuRepair is ahead of the strongest baseline in eight, tied in two, and behind in three.

We agree with the reviewer's caveats. Several families have only one or two tasks, and 12 of the 25 semantic tasks come from the seven-qubit Steane-code family; the revised Section 5.1 states both.

W3/Q3: KL-Divergence Criterion and Threshold Sensitivity

Our evaluation targets behavioral correctness under the specified measurement-output contract, not full quantum-state or unitary equivalence; a physically irrelevant global phase, for example, should not count as a fault. The 0.05 cutoff follows QuanBench+ (Slim et al., 2026), which reports a 99.7th-percentile KL divergence of about 0.048 across repeated finite-shot executions of correct reference programs. All primary evaluations use 16,384 shots and a fixed seed. The same cutoff is used at both stages: an executable mutant is retained only if its KL divergence from the clean reference exceeds 0.05, and a repair is accepted only if its divergence falls below 0.05 and it passes the structural and execution checks.

Threshold sensitivity (new, Section 5.2). We re-evaluated the saved repairs at four cutoffs (608 instances per method):

KL cutoffQBugLMQuRepairΔ (pp)
< 0.0146.38%62.01%+15.63
< 0.02546.38%62.83%+16.45
< 0.0547.0%62.83%+15.83
< 0.1047.0%62.83%+15.83

We agree with the reviewer's point: a 0.05 cutoff does not bound each outcome probability to within a few points. In practice, at the original seed all 382 accepted QuRepair repairs have KL below 0.025, and 377 below 0.01, a cutoff that rejects the reviewer's example (though it still admits about a 7-point shift on a balanced bit). Conversely, even at 0.10, looser than the cutoff used to retain faults, no retained faulty program is accepted.

Seed sensitivity (new). Re-evaluating all 608 saved QuRepair outputs under five fresh seeds, with the same programs, archived reference distributions, and 16,384 shots, gives 60.20–60.86% (mean 60.59%), 2.24 points below the original seed. QBugLM was not re-evaluated; the lowest fresh-seed value is still 13.2 points above its original-seed 47.0%. We did not vary the shot count.

We therefore treat the cutoff as an empirical tolerance, not a guarantee of distributional or unitary equivalence. The revised Sections 4.6 and 6 state this scope: differences not exposed by the tested measurement bases, inputs, or parameters may go undetected. Section 6 names exact output-state comparison and formal equivalence checking of quantum circuits as future evaluation criteria.

W4/Q4: Debugging Procedures and Inference Budgets

We agree that the primary evaluation is a system-level comparison, not a budget-matched experiment, and we did not rerun the baselines under matched budgets. All methods share the benchmark instances, the GPT-4o repair model, and the hidden evaluator, and each baseline keeps its defining workflow, adapted to four frameworks (Section 3, Appendix B). Three results bear on budgets:

  • The strongest baseline spends more. QBugLM also localizes before repairing and allows five attempts. Against it, QuRepair uses fewer model calls (3,745 vs. 12,267) and tokens (26.29M vs. 71.36M), Jev included (Appendix J), at 73% lower recorded cost, and repairs 96 more instances (382 vs. 286 of 608). One difference favors QuRepair: QBugLM receives only a binary verdict and no runtime exceptions (Appendix B.5), whereas QuRepair also receives feedback on runtime failures (Appendix A); neither sees output distributions or divergence values.
  • The advantage does not need Jev. Variants without Jev, in which GPT-4o does all the generation as in every baseline, still exceed QBugLM: 370/608 without any diagnosis stage (\41.56)and358/608withaGPT−4odiagnoser(41.56) and 358/608 with a GPT-4o diagnoser (\\83.94), against $139.00 for QBugLM (Part 1).
  • Information versus procedure. The component ablations hold the procedure fixed (five attempts, same feedback) and vary only the information. Removing the quantum-semantic layer costs 45 net jobs (p < 0.001), so quantum-specific information has an effect beyond the repair procedure. It does not explain the whole 96-job margin, however: without the semantic layer QuRepair still repairs 337/608, 51 more than QBugLM, and each other removal changes the net count by at most 8 jobs. We have not isolated that remainder; it may come from the other channels jointly (we removed them one at a time) or from the repair procedure itself.

Recorded inference costs on the 608 instances (observed spend, not imposed budgets; QuRepair's figure includes Jev's charges):

MethodRepair successRecorded cost
NL-Debugging12.8%$60.84
Self-Debugging21.1%$32.44
LDB35.0%$28.77
RING (Top@3)45.7%$8.06
QBugLM47.0%$139.00
QuRepair62.83%$37.57

Self-Debugging and LDB record somewhat lower costs than QuRepair but repair far fewer instances. RING is the cheapest: it generates 10 candidates, and we evaluate the top three. The original RING paper reports pass@1, pass@3, and pass@50; scoring only the top three is our adaptation choice (Appendix B.3). This budget differs from QuRepair's five feedback-guided attempts, so we do not claim a matched comparison with RING.

W5/Q5: Generalization and Statistical Reliability on Bugs4Q-Robust

We agree on both points. The margin is not statistically reliable on its own: QuRepair repairs 21/29 tasks (72.4%; Wilson 95% CI 54.3–85.3%) and LDB, the strongest baseline, 18/29 (62.1%; 44.0–77.3%). With a net difference of three tasks, an exact two-sided McNemar test cannot give p below 0.25 for any split of the discordant tasks. The revised Section 5.3 calls the margin too small to be conclusive on its own, and the abstract now gives the counts.

Bugs4Q-Robust answers a different question from the controlled benchmark: whether QuRepair works on independently collected, naturally occurring Qiskit defects that our mutation operators did not generate. We retain the 29 tasks whose buggy program fails and whose corrected program passes the official test, and each repair is judged by that test rather than our KL criterion. The set is Qiskit-only, and Section 6 states that it provides no equivalent real-bug evidence for Cirq, PennyLane, or OpenQASM. We therefore claim transfer to real Qiskit bugs, not statistical superiority or cross-framework real-world generalization.

We hope these results and clarifications address the reviewer's concerns, and we would be grateful if the reviewer would consider updating the assessment in light of them.

Reviewer_JKLHOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

QuRepair combines quantum-specific static analysis, API evidence, circuit representations, and a structured diagnosis model to guide iterative LLM repair across four quantum frameworks. It reports 62.83% repair success versus 47.0% for the strongest baseline, with additional evaluations on real-world bugs and different repair models.

Critical strengths

The matched fault-injection design is valuable: corresponding faults are validated across four frameworks, reducing the confounding between framework choice and bug composition.

The evaluation covers five baselines, four repair models, and an independent real-bug benchmark. The substantial improvements on Cirq and PennyLane suggest practical value beyond the canonical Qiskit setting.

Critical weaknesses

The contribution of structured diagnosis is not isolated. There are no component ablations separating Jev, static diagnostics, API evidence, and circuit rendering. Moreover, the cross-model comparison uses a single-attempt direct-repair baseline, so its improvements also reflect additional iterations and feedback.

The correctness criterion is limited to matching the configured measurement distribution below a KL-divergence threshold. This does not establish correctness under other inputs or parameters, and can miss phase differences invisible to the measured observable. The paper does not quantify how often accepted repairs retain such errors.

Questions for authors

With iterations, feedback, and inference budget held constant, how much improvement remains when Jev is removed and the same evidence is passed directly to the repair LLM?

2 comments

We thank the reviewer for recognizing the strengths of our matched cross-framework evaluation. We address the concerns below and have incorporated the additional findings into the revised manuscript.

W1/Q1: Structured Diagnosis and Component Ablations

Q1 (Jev removed, with the same evidence, iterations, and feedback). We removed the diagnosis stage entirely and supplied the same evidence directly to GPT-4o. Evidence channels, five attempts, feedback, evaluation seed, and protocol remained unchanged. This configuration repairs 370/608 instances (60.86%). For comparison:

  • Full QuRepair as reported in the paper repairs 382/608 (62.83%).
  • The reported 382/608 successful repairs come from one complete evaluation across all four frameworks. This original full-configuration run also serves as the control for the component-ablation experiments and is 96 repairs (15.8 percentage points) above QBugLM.

Relative to the reported 382/608, removing Jev lowers success by 12 instances (1.97 percentage points). We do not report a paired significance test for this comparison because aligned job-level outcomes for the no-diagnosis run were not available in the audited archive.

Inference budgets were not equalized, but the imbalance favors the no-diagnosis configuration. Full QuRepair has 9.6% lower recorded inference cost ($37.57 vs. \$41.56) and 24.6% lower average time per instance (35.88 vs. 47.60 s). Jev's typed diagnosis thus mainly improves efficiency: even without any diagnosis stage, the pipeline repairs 84 more instances than QBugLM (370 vs. 286).

Component ablations. We compared each component ablation with the original full-configuration run (382/608, same code revision) on the same 608 jobs, using exact McNemar tests on discordant jobs and paired Wald 95% confidence intervals:

 

Removed component (Jev kept)Repaired / 608Full-only / ablated-only jobsDrop vs. full run, pp [95% CI]Exact McNemar p
None (four-framework run)382———
Quantum-semantic layer33774 / 297.40 [4.18, 10.62]< 0.001
API evidence37838 / 340.66 [−2.08, 3.39]0.724
Circuit rendering37437 / 291.32 [−1.30, 3.93]0.389
Linter37742 / 370.82 [−2.04, 3.69]0.653

Only removing the quantum-semantic layer has a statistically distinguishable effect. The other three removals change success by at most 8 instances, and their confidence intervals include zero, so we do not claim separate contributions for them. Removing the semantic layer also changes the rules supplied to Jev, so it removes the semantic evidence pathway as a whole.

Diagnosis-stage alternatives. Replacing Jev with a GPT-4o diagnoser repairs 358/608 (58.88%) at 2.2× the recorded cost (\83.94vs.83.94 vs. \\37.57), fewer than with no diagnosis stage at all. Even this GPT-4o-only configuration repairs 72 more instances than QBugLM (58.88% vs. 47.0%) at lower cost ($83.94 vs. \$139.00), so QuRepair's advantage over the strongest baseline does not depend on Jev.

Finally, as Section 5.3 states, the cross-model comparison with single-attempt direct repair reflects the full iterative procedure rather than the evidence alone. The ablations above hold iterations and feedback fixed.

W2: Correctness Criterion and KL-Divergence Threshold

Our evaluation assesses correctness under the specified measurement-output contract, not full quantum-state or unitary equivalence. Repair acceptance requires D_KL < 0.05, alongside execution and structural checks. This threshold follows QuanBench+ (Slim et al., 2026), which reports a 99.7th-percentile KL divergence of approximately 0.048 from repeated finite-shot reference executions. Our primary evaluation uses 16,384 shots and a fixed seed.

For executable mutants evaluated through measurement distributions, fault inclusion requires KL divergence from the clean reference to exceed 0.05, ensuring detectable behavioral deviation under the evaluation contract. API-misuse mutants are validated separately by requiring the clean implementation to succeed and the mutated implementation to fail for the intended API-related reason.

KL-threshold sensitivity. We reevaluated saved repairs at four cutoffs on the same benchmark (608 instances per method):

KL cutoffQBugLMQuRepairΔ (pp)
< 0.0146.38%62.01%+15.63
< 0.02546.38%62.83%+16.45
< 0.0547.0%62.83%+15.83
< 0.1047.0%62.83%+15.83

Tightening the threshold from 0.05 to 0.01 reduces QuRepair's success by only 0.82 pp (five cases), while preserving an advantage exceeding 15 pp. We also re-evaluated the retained injected faulty programs: none becomes accepted at any tested cutoff, including 0.10.

Simulator-seed sensitivity. We reevaluated all 608 saved QuRepair outputs using five fresh seeds, keeping the repaired programs, archived reference distributions, and 16,384 shots fixed. Success ranged from 60.20% to 60.86% (mean 60.59%), compared with 62.83% under the original seed, an average decrease of 2.24 pp.

These experiments show limited variation across the tested thresholds and fresh seeds, although the original-seed result is 2.24 pp above the fresh-seed mean. They do not establish equivalence under untested inputs, parameters, or measurement bases, nor do they quantify undetected phase differences among accepted repairs. The revised Sections 4.6 and 6 state these limitations explicitly. Section 6 names exact output-state comparison and formal equivalence checking of quantum circuits as future evaluation criteria.

Presentation

We have revised the manuscript to incorporate these additional findings:

  • Section 3 (Debugging Methods) and Figure 1 clarify the distinct roles of evidence generation, Jev diagnosis, and GPT-4o repair.
  • Section 4 (Experimental Setup) now explains the KL-divergence threshold, mutation-validation criteria, and limitations of measurement-based correctness.
  • Section 5.2 (RQ2) includes:
    • the run provenance of the reported 382/608;
    • the component and diagnosis-stage ablations, with a paired significance analysis for the component ablations against the original full-configuration run (382/608); the full tables are in Appendix K;
    • their effectiveness and inference-cost comparisons;
    • the new KL-threshold and simulator-seed sensitivity analyses.
  • Section 6 (Limitations) now distinguishes measurement-output correctness from full quantum equivalence and acknowledges the limitations of the component ablations and unmatched inference budgets.

We hope these revisions and additional experiments address the reviewer's principal concerns, and we would be grateful if the reviewer would consider updating the assessment in light of them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.