acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking the Quantum Leap: Auditing LLM Evaluation for Quantum Code Generation

Abstract

Large language models (LLMs) are increasingly capable of generating quantum programs, but reliable evaluation remains difficult: benchmarks differ in task composition, scoring procedures, and failure handling, while leaderboard rankings can obscure uncertainty and evaluator defects. We evaluate 15 hosted frontier, open-weight, quantum-specialized, and retrieval-augmented models across five primary quantum code-generation suites containing 530 tasks, and 10 models on QuantumBenchEval, a previously unreleased collection of 96 tasks across six quantum-computing topics. We report task-level bootstrap confidence intervals, multiplicity-adjusted paired comparisons, and audits of evaluator behavior, output validity, and shared failures. QiskitHumanEval-Standard is approaching saturation for the strongest models (98.7% pass@1), whereas its Hard variant retains substantial headroom (63.8%); across the recorded primary-suite outcomes, none of the five top-two comparisons remains significant after multiple-comparison correction. Evaluation design can have an even larger effect: correcting undisclosed exact-value checks in QuantumBenchEval's T1 topic changes two models' measured pass@1 from 0% to 94.1% without changing their outputs. Our audit also identifies 17 QuanBench-117 tasks with shared failures associated with benchmark-owned imports, while controlled probes of the current evaluator reveal unexecuted unit tests, leakage of canonical functions into candidate execution, and reference imports that can block otherwise valid candidates. These defects can affect both apparent failures and apparent passes, requiring historical QuanBench outcomes to be revalidated under an isolated evaluator. Together, these results show that quantum code-generation progress cannot be characterized by leaderboard scores alone. We release the evaluation harness, per-sample outcomes, and audit artifacts, and advocate protocols that validate scoring and test execution, isolate candidate and reference code, report uncertainty and output validity, and respect sampling assumptions when estimating pass@.

Primary area:
datasets and benchmarks
Keywords:
Quantum Computing, Benchmark, Code generation, Qiskit

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fr7LOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents benchmarking work for quantum code generation across fifteen models on five existing benchmarks and introducing QuantumBenchEval dataset. Paper evaluates code generation performance with bootstrap confidence intervals, paired comparison tests, and analysis of failure cases. The paper discusses evaluator code, uncovering issue like unexecuted test classes, leakage of reference code to candidate namespace, and over-strict grading assertions in benchmarks. Overall, the study shows current quantum code leaderboards are heavily affected by evaluation setup errors rather than pure model capabilities.

Critical strengths
  1. The paper provides deep manual audit of existing evaluator code structures, revealing serious evaluation flaws like unexecuted test classes in QuanBench-117 and implicit reference scope leakages that skew reported leaderboards. 2. The statistical rigor in benchmark reporting, including multiplicity-adjusted paired permutation tests and task-level bootstrap confidence intervals, sets very good methodological standard for code generation evaluation.
Critical weaknesses

It is not clear what are the unique features of quantum code generation that make existing classical program evaluation techniques insufficient beyond standard execution asserts.

Questions for authors
  1. Section 3.1 mentions temperature 0.8 is used where exposed, but Claude Fable 5.1 and Muse-Glimmer-30B use 1.0, and token limits vary between 2,048 and 32,768. Could the paper add uniform generation setting experiments to ensure cross-model comparisons are not confounded by generation budget? 2. Section 4.5 describes QuanBench findings like unit-test classes not executing. Can the paper clarify how many specific tasks in QuanBench-117 flip their final pass/fail label when re-run under fully isolated test runner? 3. Section 4.4 and Figure 2 discuss QBE topic evaluator corrections. Can the paper provide algorithmic flow or detailed pseudocode for how test contracts are formally defined and verified without checking unstated constraints? 4. In Section 3.4 and Table 3, Claude Fable 5.1 shares provider with the judge model on QCoder and QBE T3. Could the paper conduct cross-validation with an independent non-provider judge to quantify potential self-preference bias?
3 comments
AuthorsOpenReview ↗

Response to Reviewer fr7L

Thank you for the careful reading and the specific, actionable questions. Our general comments (1/2, 2/2) report the saved-answer replay and per-model results (Table R1).

Weakness. What is quantum-specific

Classical execution assertions remain the right machinery, but the oracle must encode quantum semantics: state/operator equivalence up to global but not relative phase, qubit and bit ordering, measurement mappings, clean-ancilla behavior, finite-shot uncertainty, and gate/depth/qubit constraints. For example, matching measurement probabilities cannot detect every relative-phase error, and our audit found a historically accepted answer with exactly this error. Our contribution is specifying and validating such oracles inside standard testing machinery, not claiming that classical testing is inherently inadequate. We will make this framing explicit in the revision.

Q1. Uniform generation settings

We agree that differing output caps and temperature policies confound cross-model comparison. A retrospective audit of 38,975 existing-suite and 4,725 QBE answers found missing usage/stopping records and cases where a configured temperature was not sent, so filtering saved outputs by length would not be a controlled experiment. We are running a controlled experiment: output caps of 2,048 and 16,384 tokens, five answers per condition on replay-calibrated tasks, scored with the repaired runner, with stop reason and output length logged per answer. GPT-5 is included, because many of its historically passing answers are empty (general comment 1/2). A common cap does not equalize hidden reasoning budgets across providers, and we will note this. Results will follow in a comment.

Q2. QuanBench-117 label flips

On the 102 replay-calibrated tasks, we define a model–task pass as at least one passing answer among five, a failure as all five failing, and indeterminate otherwise. Under the isolated runner, 290 model–task pairs flip pass→fail and 40 flip fail→pass, spanning 87 of the 102 tasks; another 582 become indeterminate. These are model–task transitions, not distinct tasks. Per-model rates are in Table R1.

Q3. Contract definition procedure

1. Freeze candidate-facing prompt, inputs, interface and source revision
2. Map each assertion to an explicit or mathematically implied requirement
3. Hold contradictory or unresolved requirements instead of choosing silently
4. Define oracle, tolerances, phase policy and disclosed resource constraints
5. Validate positive alternatives and targeted incorrect/shortcut controls
6. Cross-check independently computable quantities with a separate implementation
7. Freeze contract, environment, seeds, scoring rules and coverage denominator
8. Replay unchanged answers only where the disclosed interface supplies evidence
9. Retain unresolved outcomes; version all subsequent changes

This is an auditable procedure rather than a formal soundness proof. Step 2 is what prevents checking unstated constraints: an assertion without a stated or mathematically implied requirement is removed or held. For example, a returned T6 partition supports independent metric checks, but it cannot retroactively become a required output when the prompt asked only for scalar metrics. We will add this procedure to the revision.

Q4. Independent judge

We are running Muse-Glimmer-30B (non-Anthropic) as an additional judge on a frozen sample of 344 saved answers (210 QCoder, 134 QBE T3), with historical labels withheld. Muse is also an evaluated model, so it is not fully independent; its own answers will be analyzed separately as self-judgments. We will report calibration first, then agreement and any provider-specific differences, including for Claude Fable 5.1. For T3, only the references with calibrated energy targets (5 of 15 contracts) support numerical scoring. Because cross-judge agreement alone does not establish correctness, disagreements will be checked against executable evidence.

AuthorsOpenReview ↗

Update on Q1 and Q4

Brief pointers to new results.

Q1: we checked per model whether the output cap bound generation and regenerated where it did (Follow-up 1). This is narrower than the uniform grid we first described (sorry, we did as much as we could in the time to give you a reasonable response), but it targets the actual confound. The cap bound GPT-5 and Gemini; two unaffected controls (Opus 4.6, Mistral-24B) do not change at a larger cap. Temperature values as actually sent are also reported.

Q4: the non-provider judge study is complete (Follow-up 2). Disagreements were checked against the official QCoder execution backend rather than resolved by assuming either judge is correct. Muse's own answers are reported separately, and four self-judging disagreements are too few to quantify self-preference, so we make no bias claim.

Looking forward to further discussion!

I would like to thank the authors for the responses. In light of the rebuttal and other reviews, I keep my score.

Reviewer_HANXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigate the performance of LLM in generating quantum computing programs. This paper compares several fromtier models as well as open-weight models on multiple existing quantum program datasets. One contribution is that the authors create a new dataset for better comparison. Another important contribution is to check whether or not the benchmark itself is actually valid.

Critical strengths
  1. The authors discovered that there are several issues in existing benchmarks. For examples, some unittest are not actually carried out, and some incorrect generated outputs actually get a "pass". The authors demonstrated several real examples in the paper.

  2. The authors comprehensively compare frontier models and open-weights models. To the reviewer's knowledge, most quantum programming benchmarks are investigated.

  3. Overall, the statistical methods used in the paper is appropriate.

Critical weaknesses
  1. Hard Benchmarking not necessarily "hard in reasoning". QiskitHumanEval-Hard does not mean that it requires more quantum reasoning capability. It simply provides less python code hints.
  2. The proposed QBE is not very robust. In the Appendix, it is mentioned that some of the problems have self-inconsistencies. And certain BAD answers can pass all checks.
  3. LLM comparisons are not entirely fair. The output token usages are not consistent across tests. For example, the QCoder. While the authors mentioned this limitation, it is unclear whether such issues have been resolved.
  4. Fundamentally, the work is still following those existing works: measuring PASS/FAIL rates. It is not clear why certain model fails some tests. For example, is it not familiar with the Qiskit API? Or not understanding the concept of quantum state, Hamiltonian or ansatz design? Without these analysis, I am afraid that the work provides limited insights to LLM-guided quantum programming.
Questions for authors
  1. LLM judge is not reliable. In Appendix E, the authors actually show some examples. So what is the plan to resolve this problem?
  2. The benchmark is not valid. Since there are identified evaluator defects, then why the QuanBench results are not fully re-evaluated using the corrected evaluator?
  3. What is the insight os certain LLM fail on some quantum tasks?
2 comments
AuthorsOpenReview ↗

Response to Reviewer HANX

Thank you for distinguishing evaluator reliability from model capability; our follow-up supports that distinction. Our general comments (1/2, 2/2) report the saved-answer replay and per-model results (Table R1).

W1. Meaning of "Hard"

Agreed. QiskitHumanEval-Hard reduces supplied scaffolding; it does not establish greater quantum-reasoning difficulty. We will describe it operationally and will not interpret its score differences as a measure of quantum reasoning.

W2. QBE robustness

We agree, and we applied the same audit procedure to QBE that we applied to the existing suites. We reproduced the loophole you note with controlled fabricated-metric and invalid-partition answers across all 17 T6 tasks; both the original and our earlier repaired tests accept them. Because the original T6 interface did not require returning a partition, saved answers cannot be uniformly rescored, and we will not claim corrected T6 scores for them. For the self-inconsistencies, an audit of all 15 T3 reference contracts finds 5 with calibrated energy targets, 7 with conflicts (e.g., Hamiltonians inconsistent with molecular reference energies, hidden active-space choices) and 3 conditional on interpretation.

We will release a versioned QBE v1.1 alongside the original: T6 tasks will require the partition so that the new consistency check (which passes 255 synthetic controls) applies, and T3 contracts will be corrected where the intended reference can be made explicit, or otherwise removed. The revision will scope v1 results to the evidence above.

W3. Generation fairness

Agreed, this is not yet resolved. Our settings audit also found that QCoder lacks per-answer token-usage records and that some configured temperatures were not sent. A controlled uniform-budget experiment is in progress (two output caps, stop reason and output length logged per answer, scored with the repaired runner), we will post results as a follow-up. Until then, comparisons are stated as conditional on recorded settings.

W4 / Q3. Why models fail

The replay identifies concrete mechanisms beyond pass/fail. The clearest is empty output: most of GPT-5's historical decline comes from empty answers that the old evaluator passed (general comment 1/2), plausibly linked to its output budget. Inspection also finds Qiskit API errors (importing unavailable or removed symbols), circuit-construction errors (duplicate control/target qubits, invalid classical-bit destinations), missing returns, and quantum-semantic errors (e.g., a wrong relative-phase preparation that a historical judge accepted). Our planned breakdown of indeterminate outcomes will separate Qiskit API/version drift from other causes, which directly addresses your question of API familiarity versus quantum understanding. The examples above distinguish these mechanisms but are not yet a prevalence estimate.

Q1. Judge reliability: plan

(i) Official QCoder evaluation runs through a rate-limited platform endpoint, so official agreement is unmeasured, we will state this as a limitation. A local executable check on five tasks agrees with the historical judge on 250/284 determinate answers (88.0%). (ii) We are running Muse-Glimmer-30B as a second, non-Anthropic judge on a frozen sample of 344 saved answers. It must pass calibration first, malformed and unresolved cases are retained, and Muse's own answers are analyzed separately. (iii) Disagreements will be interpreted against executable checks, not by assuming either judge is correct. Results will follow in a comment.

Q2. Full QuanBench re-evaluation

We have now re-evaluated the saved QuanBench answers under the corrected, isolated runner (Table R1). The replay covers the 35/44 and 102/117 tasks whose reference solutions pass the corrected run, we did not score tasks where the reference itself fails, since a failing reference cannot validate candidates. The replay repairs execution, not every semantic contract: some original tests still conflict with their prompts, which is why we report indeterminate outcomes separately rather than counting them as failures.

AuthorsOpenReview ↗

Update on Q1, Q3/W4 and W3

Brief pointers to new results.

Q1 (judge plan): carried out. An independent open-weights judge and the official QCoder execution backend are in Follow-up 2; the official backend accepted 19 of 25 answers the original judge rejected, and most official rejections were platform-rule violations a text-only judge cannot check. A judge-version inconsistency we found and corrected is in Follow-up 3.

Q3/W4 (why models fail): an error-type distribution over all 4,687 replay failures is in Follow-up 4: 53% assertion failures, 32% syntax errors, 12% empty answers, with strongly model-dependent mixes.

W3 (token budget): a per-model audit and regenerations are in Follow-up 1. The cap bound GPT-5 and Gemini; raising it roughly doubles GPT-5's corrected QuanBench scores, while unaffected models do not change. We would welcome any remaining concerns, but hopefully we have addressed your points.

Reviewer_MVvsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper evaluates LLMs on quantum code-generation benchmarks and examines the reliability of their evaluation procedures. It compares 15 models across five existing suites and introduces QuantumBenchEval, a 96-task benchmark. The analysis combines uncertainty estimates with evaluator audits, identifying scoring defects that can produce both false passes and false failures.

Critical strengths

The evaluator audits address a practically important issue. The examples of unexecuted tests, reference-code leakage, and undisclosed scoring requirements demonstrate how benchmark scores can misrepresent model performance. Reporting uncertainty and explicitly acknowledging evaluation limitations also improves the interpretation of the results.

Critical weaknesses
  1. The corrected evaluations require further validation. Historical QuanBench outputs have not been replayed under a corrected, isolated evaluator, leaving the extent and model-dependent impact of the identified defects unresolved. For QBE, removing undisclosed requirements addresses false failures, but Appendix C.5 documents remaining semantic loopholes, including an invalid T6 partition that passes all checks. Consequently, the substantial increases after rescoring do not by themselves establish corresponding improvements in measurement accuracy. The audit findings are useful, but the model comparisons and proposed benchmark need stronger validation.

  2. Evaluation provenance and reported results are not fully consistent. The submission describes QCoder as lacking executable tests, whereas the cited QCoder paper presents a simulator-based evaluator. The reason for replacing this procedure with an LLM judge, and the agreement between the two approaches, remain unclear. Separately, Table 2 reports QuanBench-44 scores of 31.4% for LLaMA-3.1-8B and 15.9% for Quantum RAG, while Appendix D.1/Table 9 states that these outputs duplicated QuanBench-117 generations and were not evaluated on QuanBench-44. These inconsistencies make it difficult to determine which measurements support the subsequent analyses.

Questions for authors
  1. Can you report corrected replay results for the saved QuanBench outputs and assess how addressing the QBE semantic loopholes affects the reported scores and conclusions?
  2. Which QCoder dataset version and evaluation protocol were used? Why was the simulator-based evaluator replaced, and how well does the LLM judge agree with execution-based evaluation?
  3. Which QuanBench-44 results are authoritative? Were the duplicated runs excluded consistently from statistical comparisons, rank correlations, and coverage analyses?
2 comments
AuthorsOpenReview ↗

Response to Reviewer MVvs

Thank you for separating the detection of evaluator defects from validation of their effect on the comparisons. Our general comments (1/2, 2/2) report the saved-answer replay, per-model results (Table R1) and the mechanism behind the largest shift. We address your questions below. We have tried to answer everything, and have things running, which we will update as soon as we can (projected run time is ~30 hours).

1. Replay and effect on conclusions

The replay covers the 35/44 QuanBench-44 and 102/117 QuanBench-117 tasks whose references passed runner calibration. The effects are model-dependent (Table R1), so the historical comparisons cannot be retained by acknowledging the defects alone; we will report the replay results alongside them and scope comparisons to the selected tasks. Separately from execution repair, a matched repair of six semantic contracts on 420 saved answers yields 172 passes versus 72 under the original tests with the same repaired runtime. This shows contract sensitivity on selected tasks; it is not a replacement ranking.

QBE semantic loopholes. We agree that the earlier score increases do not demonstrate improved measurement accuracy. Across all 17 T6 tasks, both the original tests and our earlier reproducibility-only repair accept two controlled counterexamples: fabricated metrics without the requested computation, and fabricated metrics with an invalid partition. A new check of returned partitions against their reported metrics passes 255 synthetic controls, but the historical interface did not require returning a partition, so we cannot fairly fail saved answers for omitting one. Of 850 preserved T6 records, 129 return explicit partitions with consistent metrics; the rest omit partitions or lack complete return evidence. We will interpret the earlier increases as sensitivity to scoring requirements and will not claim corrected T6 scores or QBE-wide semantic validity.

2. QCoder version and protocol

You are correct; describing QCoder as lacking executable evaluation was wrong. We used all 67 problems in the QCoder repository at https://github.com/QCoder-Bench/QCoder-Benchmark, which match the publisher dataset revision da10cf5c0d04298bf27ee33e4e110ad205a4eac5. The repository contains task files and a generation/judging script but no local evaluator: official validation submits each answer to an authenticated, rate-limited QCoder platform endpoint (100 validations per 24 hours), within up to five simulator-feedback refinement rounds. Scoring every saved answer this way was impractical at our scale, so we used an LLM-judged adaptation. Because the protocol and task set (the QCoder paper reports 58 problems) both differ, we will label it as an adaptation and not compare with the QCoder paper's accuracies.

Agreement with the official evaluator remains unmeasured. As a local proxy, an executable state-consistency check on the five QPC001_A1–A5 tasks (350 saved answers) agrees with the historical judge on 250/284 determinate outcomes (88.0%); it does not verify the official depth, gate or qubit constraints. The saved runs also used two judge versions: Sonnet 4 for GPT-5, Gemini 3 Flash, Gemma-3-4B, Granite-Qiskit, both Mistral models and Qwen-Qiskit; Sonnet 4.6 for the remaining seven. Judge version is therefore confounded with model identity; we will disclose this and avoid cross-group QCoder comparisons. An additional Muse-Glimmer-30B judge is running (general comment 2/2).

3. QuanBench-44 entries

The LLaMA-3.1-8B and Quantum RAG QuanBench-44 entries are not supported measurements and will be marked as such and excluded from all analyses; their QuanBench-117 runs are retained. All 220 status records in each run match the corresponding QuanBench-117 subset, and independent QuanBench-44 generation provenance could not be established. We recomputed the statistical comparisons, rank correlations and coverage analyses with these exclusions applied consistently; 1,413 of 1,494 pairwise comparisons remain, with the multiple-comparison adjustment recomputed. We will check each affected conclusion against both this exclusion analysis and the replay results.

AuthorsOpenReview ↗

Update on Q1–Q2

Brief pointers to new results.

Q1: GPT-5 answers regenerated at a 16,384-token cap and scored with the same frozen evaluator as Table R1 rise from 26.1–32.2% to 54.3–63.3% (QuanBench-117) and from 25.7–28.0% to 52.0–54.9% (QuanBench-44); Opus 4.6, which the cap barely affected, is unchanged (Follow-up 1).

Q2: agreement with the official QCoder execution backend is now measured on 40 selected answers. On the 26 cases where two LLM judges disagreed, the official verdict sided with the independent judge in 20 (Follow-up 2). Re-judging the Sonnet-4 group with Sonnet 4.6 showed the two historical judge versions differ about threefold in strictness; a single-judge QCoder table is in Follow-up 3. We would welcome any remaining concerns.

Comments on the paper

7 comments
AuthorsOpenReview ↗

General Response to All Reviewers: Evaluator Replay and Updated Findings (1/2)

We thank the reviewers, it seems we got valuable feedback from all three. We are excited to start the discussion period and open with a general comment to all reviewers.

All three reviews ask whether the evaluator defects we identified were validated against the reported results. We have now replayed saved answers under a repaired runner. The replay supports the paper's central claim: benchmark outcomes are materially driven by evaluator behavior, and the effect is model-dependent. We are narrowing the paper accordingly. The evaluator audit and its consequences for interpreting quantum code-generation benchmarks are the primary contribution. Cross-model comparisons are conditional on recorded generation settings, task selection and unresolved outcomes.

Replay setup.

Unchanged saved answers were re-executed under a versioned runner that separates reference and candidate namespaces, executes unittest methods, rejects zero-check passes and implements the numerical helpers. Original test assertions were retained. We replayed the 35/44 QuanBench-44 and 102/117 QuanBench-117 tasks whose references passed runner calibration before any saved answer was replayed (685 reference/seed checks and 22 runner controls passed). The remaining tasks' references did not pass calibration, including one whose earlier pass depended on an alias patch absent from this runner. Excluded tasks had lower historical pass rates, so selected-task rates are higher than full-suite rates. The two QuanBench-44 runs without established generation provenance (LLaMA-3.1-8B, Quantum RAG) were excluded; their QuanBench-117 runs are retained.

Results.

Of 9,925 replayed answers, 2,997 pass, 4,687 fail and 2,241 (22.6%) are indeterminate: outcomes that cannot be attributed to the candidate because of unresolved environment, interface or prompt/test conflicts. Relative to historical flags, 2,483 passes become failures, 358 failures become passes and 978 passes become indeterminate. The runner changes several features together, so we do not attribute transitions to individual defects. Per-model results are in Table R1 (next comment). Preliminary inspection suggests many indeterminate outcomes involve Qiskit API and version drift (e.g., imports removed in later releases). We will classify each cause, re-run affected answers in an environment pinned to the benchmark's Qiskit version, count imports of nonexistent modules as failures, and report the breakdown in a follow-up comment.

GPT-5.

The largest shift has a concrete mechanism. On the selected tasks, 75/175 (QuanBench-44) and 258/510 (QuanBench-117) saved GPT-5 answers are empty strings that carried historical pass flags; the replay scores them as failures. This is consistent with the reference-fallback defect: the historical runner executed the reference first in shared globals, so an empty candidate could inherit the reference implementation. Historical result files do not cryptographically bind each flag to its generated text, so per-answer attribution is not proven. We will report an empty-candidate control under the historical runner. The empty outputs plausibly also reflect GPT-5's restrictive historical output budget. GPT-5 is included in our uniform-budget experiment, and we will report its regenerated results alongside the historical run. On its remaining answers, GPT-5's replay pass rate is 45–49% (QuanBench-44) and 53–65% (QuanBench-117). Its historical-to-replay decline is therefore largely explained by evaluator acceptance of empty answers and generation budget, not by a measured difference in code quality.

AuthorsOpenReview ↗

Evaluator Replay and Updated Findings (2/2)

As mentioned in our initial comment, Table R1 reports per-model results from the selected-task replay..

Table R1

Selected-task replay. "Pass" counts indeterminate answers as failures; "Indet." is the indeterminate share, so the most favorable possible pass rate is Pass + Indet. Δ = Pass − historical, in percentage points; because Pass treats indeterminate answers as failures, Δ is the largest possible decline, and the most favorable change is Δ + Indet. These are identification bounds on saved answers, not confidence intervals (unlike the bracketed values in Table 2).

ModelQB-44 hist.QB-44 passΔQB-44 indet.QB-117 hist.QB-117 passΔQB-117 indet.
GPT-6 Astra62.962.3−0.60.091.263.7−27.55.1
Claude Fable 5.169.770.3+0.60.091.267.5−23.72.5
Gemini 3.1 Pro Preview66.361.7−4.60.089.465.9−23.57.6
Claude Opus 4.660.058.9−1.10.690.255.9−34.39.4
Muse-Glimmer-30B53.745.1−8.68.088.045.9−42.213.3
Gemini 3.6 Flash38.934.9−4.010.355.135.9−19.213.7
Gemini 3 Flash30.929.1−1.718.947.328.4−18.814.9
GPT-570.925.7−45.12.386.326.1−60.26.1
Mistral-Qiskit30.323.4−6.922.985.317.5−67.828.4
Qwen-Qiskit32.021.1−10.925.780.815.1−65.732.7
Granite-Qiskit10.93.4−7.426.333.71.2−32.530.6
Gemma-3-4B12.08.6−3.441.182.95.7−77.345.1
Quantum RAGexcl.excl.–excl.45.94.7−41.256.1
LLaMA-3.1-8Bexcl.excl.–excl.30.20.6−29.616.1
Mistral-7B6.31.1−5.177.711.80.8−11.077.5

On QuanBench-44, frontier-model replay rates are close to historical, except for GPT-5 (which we have agreed to re-run). The large shifts are on QuanBench-117, where historical unit-test classes did not execute, and in smaller open-weight models. For example, Gemma-3-4B's historical QuanBench-117 rate (82.9%) was close to frontier models' and falls to 5.7% under replay. Indeterminate outcomes concentrate in smaller models; their cause breakdown will follow as described in our previous comment.

Other corrections

(i) QCoder provides simulator-based evaluation through an authenticated, rate-limited platform endpoint. Our results use an LLM-judged adaptation, which we will label as such, with two recorded judge versions (details in our reply to MVvs). (ii) The two unsupported QuanBench-44 entries in Table 2 will be marked as unsupported and excluded from all analyses. (iii) Generation budgets were unequal across models; a uniform-budget experiment is in progress. (iv) We are running Muse-Glimmer-30B as an additional, non-Anthropic judge on a frozen sample of 344 saved answers. Because Muse is also an evaluated model, its own answers will be analyzed separately as self-judgments. We will post results from (iii) and (iv) as follow-up comments.

Planned revisions

Retain the published QuanBench results as the historical record and report the selected-task replay, Table R1 and the exclusions alongside them; correct the QCoder protocol and judge-version descriptions; scope QBE validity, generation-setting and model-comparison claims to the evidence; add the contract-definition procedure and quantum-specific oracle examples.

AuthorsOpenReview ↗

Follow up 1: Generation Budget (fr7L Q1, HANX W3)

As promised in our earlier comments we report a generation budget analysis. The design differs from the one we described: instead of a fixed caps-by-tasks grid, we checked for every model whether the historical output cap actually constrained generation, and regenerated with a larger cap only where it did, plus two unaffected models as controls. Published Table 2 values are unchanged; these are additional analyses.

Where the cap bound. GPT-5 (2,048 cap): 1,183 of 2,315 non-QCoder answers stop at exactly 2,048 output tokens, and 1,182 of those contain no code (vs 3 of 1,132 below the cap); on QCoder, 211 of 335 are empty. Gemini (QCoder): about a third of sampled answers end mid-expression despite short visible text, consistent with hidden thinking consuming the cap. Opus 4.6: 41 of 2,315 answers (1.8%) reach the cap. Open-weight models: no saved answer approaches the cap. LLaMA-3.1-8B's empty answers are formatting failures, not truncation. Fable 5.1, GPT-6 Astra and Muse already had 16,384–32,768.

Regenerations (only the cap changed; single regeneration per model):

ModelSuiteHistorical cap16,384 cap
GPT-5QiskitHumanEval46.596.2
GPT-5QiskitHumanEval-Hard24.143.7
GPT-5QuanBench-117 (102 tasks)†26.1–32.254.3–63.3
GPT-5QuanBench-44 (35 tasks)†25.7–28.052.0–54.9
GPT-5QCoder‡29.938.5
Gemini 3 FlashQCoder‡5.748.1
Gemini 3.1 ProQCoder‡22.145.1
Gemini 3.6 FlashQCoder‡23.347.5
Opus 4.6 (control)QuanBench-117 (102 tasks)†55.9–65.355.5–62.5

Pass@1, %. †Same frozen evaluator and host as Table R1; ranges are identification bounds. ‡One judge (Sonnet 4.6) for both columns.

For GPT-5, 60% of regenerated non-QCoder answers used more than 2,048 tokens. Gemini's thinking expands to fill the budget (median ≈15,700 thinking tokens for 3 Flash and 3.1 Pro), and 150, 91 and 23 of 335 answers still hit 16,384, so the Gemini values are lower bounds. In the GPT-5 QCoder rerun, the runner retried 82 requests that exhausted the cap (9 slots remained empty); the historical runner saved such answers as empty, which slightly favors the rerun. A second control, Mistral-3.2-24B-Qiskit at 2,048 vs 32,768 (125 answers each), shows no answer above 898 tokens and no change (78.0% vs 76.8% of determinate outcomes on 15 supported tasks). Raising the cap does not by itself raise scores.

Temperature, as sent. 0.8 reached Gemini, Opus 4.6 and the local open-weight models. GPT-5 and GPT-6 Astra received no temperature parameter, so the OpenAI default (1) applied, including in the regeneration. Claude Fable 5.1 does not accept a temperature parameter and ran at its fixed default (1.0). Muse ran at 1.0 with top-p 0.95 and top-k 64. Section 3.1 will state values as sent.

Consequences. Comparisons involving GPT-5 (all suites) and Gemini (QCoder) will be revised or caveated, with regenerated results reported alongside the historical ones. We do not claim equal compute across providers: hidden reasoning, tokenizers and accounting differ.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.