- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents benchmarking work for quantum code generation across fifteen models on five existing benchmarks and introducing QuantumBenchEval dataset. Paper evaluates code generation performance with bootstrap confidence intervals, paired comparison tests, and analysis of failure cases. The paper discusses evaluator code, uncovering issue like unexecuted test classes, leakage of reference code to candidate namespace, and over-strict grading assertions in benchmarks. Overall, the study shows current quantum code leaderboards are heavily affected by evaluation setup errors rather than pure model capabilities.
Critical strengths
- The paper provides deep manual audit of existing evaluator code structures, revealing serious evaluation flaws like unexecuted test classes in QuanBench-117 and implicit reference scope leakages that skew reported leaderboards. 2. The statistical rigor in benchmark reporting, including multiplicity-adjusted paired permutation tests and task-level bootstrap confidence intervals, sets very good methodological standard for code generation evaluation.
Critical weaknesses
It is not clear what are the unique features of quantum code generation that make existing classical program evaluation techniques insufficient beyond standard execution asserts.
Questions for authors
- Section 3.1 mentions temperature 0.8 is used where exposed, but Claude Fable 5.1 and Muse-Glimmer-30B use 1.0, and token limits vary between 2,048 and 32,768. Could the paper add uniform generation setting experiments to ensure cross-model comparisons are not confounded by generation budget? 2. Section 4.5 describes QuanBench findings like unit-test classes not executing. Can the paper clarify how many specific tasks in QuanBench-117 flip their final pass/fail label when re-run under fully isolated test runner? 3. Section 4.4 and Figure 2 discuss QBE topic evaluator corrections. Can the paper provide algorithmic flow or detailed pseudocode for how test contracts are formally defined and verified without checking unstated constraints? 4. In Section 3.4 and Table 3, Claude Fable 5.1 shares provider with the judge model on QCoder and QBE T3. Could the paper conduct cross-validation with an independent non-provider judge to quantify potential self-preference bias?
3 comments
Update on Q1 and Q4
Brief pointers to new results.
Q1: we checked per model whether the output cap bound generation and regenerated where it did (Follow-up 1). This is narrower than the uniform grid we first described (sorry, we did as much as we could in the time to give you a reasonable response), but it targets the actual confound. The cap bound GPT-5 and Gemini; two unaffected controls (Opus 4.6, Mistral-24B) do not change at a larger cap. Temperature values as actually sent are also reported.
Q4: the non-provider judge study is complete (Follow-up 2). Disagreements were checked against the official QCoder execution backend rather than resolved by assuming either judge is correct. Muse's own answers are reported separately, and four self-judging disagreements are too few to quantify self-preference, so we make no bias claim.
Looking forward to further discussion!
I would like to thank the authors for the responses. In light of the rebuttal and other reviews, I keep my score.
Response to Reviewer fr7L
Thank you for the careful reading and the specific, actionable questions. Our general comments (1/2, 2/2) report the saved-answer replay and per-model results (Table R1).
Weakness. What is quantum-specific
Classical execution assertions remain the right machinery, but the oracle must encode quantum semantics: state/operator equivalence up to global but not relative phase, qubit and bit ordering, measurement mappings, clean-ancilla behavior, finite-shot uncertainty, and gate/depth/qubit constraints. For example, matching measurement probabilities cannot detect every relative-phase error, and our audit found a historically accepted answer with exactly this error. Our contribution is specifying and validating such oracles inside standard testing machinery, not claiming that classical testing is inherently inadequate. We will make this framing explicit in the revision.
Q1. Uniform generation settings
We agree that differing output caps and temperature policies confound cross-model comparison. A retrospective audit of 38,975 existing-suite and 4,725 QBE answers found missing usage/stopping records and cases where a configured temperature was not sent, so filtering saved outputs by length would not be a controlled experiment. We are running a controlled experiment: output caps of 2,048 and 16,384 tokens, five answers per condition on replay-calibrated tasks, scored with the repaired runner, with stop reason and output length logged per answer. GPT-5 is included, because many of its historically passing answers are empty (general comment 1/2). A common cap does not equalize hidden reasoning budgets across providers, and we will note this. Results will follow in a comment.
Q2. QuanBench-117 label flips
On the 102 replay-calibrated tasks, we define a model–task pass as at least one passing answer among five, a failure as all five failing, and indeterminate otherwise. Under the isolated runner, 290 model–task pairs flip pass→fail and 40 flip fail→pass, spanning 87 of the 102 tasks; another 582 become indeterminate. These are model–task transitions, not distinct tasks. Per-model rates are in Table R1.
Q3. Contract definition procedure
This is an auditable procedure rather than a formal soundness proof. Step 2 is what prevents checking unstated constraints: an assertion without a stated or mathematically implied requirement is removed or held. For example, a returned T6 partition supports independent metric checks, but it cannot retroactively become a required output when the prompt asked only for scalar metrics. We will add this procedure to the revision.
Q4. Independent judge
We are running Muse-Glimmer-30B (non-Anthropic) as an additional judge on a frozen sample of 344 saved answers (210 QCoder, 134 QBE T3), with historical labels withheld. Muse is also an evaluated model, so it is not fully independent; its own answers will be analyzed separately as self-judgments. We will report calibration first, then agreement and any provider-specific differences, including for Claude Fable 5.1. For T3, only the references with calibrated energy targets (5 of 15 contracts) support numerical scoring. Because cross-judge agreement alone does not establish correctness, disagreements will be checked against executable evidence.