- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(−1)
- Soundness
- 2(+1)
Read review
Paper summary
This paper introduces QASM-Eval, a training dataset and executable benchmark for OpenQASM 3 code completion beyond conventional gate-level circuits. It covers classical control, timing, pulse operations, and combinations of these capabilities through 1,200 test tasks across 100 themes, together with approximately 4,000 training examples. An extended verifier checks syntax, required constructs, measurement distributions, and execution timelines. The experimental study evaluates zero-shot generation, cross-theme few-shot prompting, a documentation-augmented agent, and LoRA supervised fine-tuning using QASM-Eval and existing quantum-programming datasets. The principal contribution is the task coverage, dataset, and evaluation infrastructure.
Critical strengths
-
The focus on OpenQASM 3 features beyond gate sequences is useful. Classical feedback, timing constraints, and pulse control introduce requirements that conventional circuit-generation tasks do not adequately test. Giving these features dedicated tasks, including tasks that combine them, is a sensible contribution to OpenQASM 3 program generation.
-
The experiments include useful controls. The authors evaluate several model families and compare fine-tuning against both same-model few-shot prompting and training on existing datasets. The theme-held-out split is also preferable to randomly separating variants of the same theme. For Llama-8B, QASM-Eval fine-tuning raises pass@1 from 32.87% with three-shot prompting to 46.93%, providing evidence that the training data helps on the covered tasks.
Critical weaknesses
- The verifier accepts some incorrect completions and rejects a valid timing alternative. Checks of the unmodified released code produced the following results. These checks used the default 8,000-shot setting, and the reference completions passed.
- In timing/theme01/v1/instance1, changing the requested rx(1.5708) to rx(0) still passes. The measurement-distribution check detects the difference, but the final decision ignores it because the timeline matches.
- In classical/theme17/v1/instance2, setting the RZ angles to zero while keeping the loop and arithmetic also passes. The resulting state has fidelity approximately 0.418 with the reference state. Their computational-basis measurement distributions nevertheless coincide, so the test misses the phase error.
- In timing/theme09/v1/instance1, the requested delay is the duration of an X gate. The released default device configuration assigns 40 ns to X, but durationof({ x q[0]; }) evaluates to 1 ns. A fixed 1 ns delay is therefore accepted, while the corresponding 40 ns delay is rejected. These failures affect what a passing score means. In the first two cases, the completion does not perform the operations specified in the prompt. The third case introduces an inconsistency between the interpreter’s duration calculation and its own device configuration. The reported 94.5% expert agreement on 200 outputs does not resolve these problems. The checks above are targeted counterexamples, so they do not establish how often such errors occur or whether model rankings change. That impact needs to be measured before the main results can be trusted as an assessment of program correctness.
- It remains unclear how much of the SFT improvement transfers to independently written tasks. The theme-held-out split is a useful precaution, but both sides still share the task-generation process and description style. Many prompts give detailed instructions for a short completion inside a supplied program. The experiments do not show how well the trained models handle specifications written independently of that process. The comparisons with QCircuitBench and Agent-Q leave a related uncertainty. Matching training token counts does not match OpenQASM 3 feature coverage or completion format. Their performance gap therefore does not isolate the benefit of the proposed construction procedure. The same-model few-shot results give a more informative comparison here. For Llama-70B, overall pass@1 increases from 55.28% to 61.17% after fine-tuning, but on Complex tasks the change is only 33.07% to 33.93%. The evidence for an SFT advantage on those combined workflows is modest. An independently authored test subset, or a closer training-data control, would help determine which gains carry beyond the benchmark’s particular format.
Questions for authors
-
Does the released verifier match the version used for the reported experiments? Please explain the expected decisions for the three cases above, especially the handling of incorrect gate parameters in Timing tasks and the implementation of durationof.
-
How do the scores change after these issues are corrected? Re-scoring saved model outputs would help establish their impact. Tests containing deliberately incorrect programs and valid equivalent alternatives would also be useful for checking whether the same problems occur elsewhere in the four categories.
-
Do you have results on tasks or descriptions written independently of the generation pipeline? If not, is there a training comparison that more closely matches the OpenQASM 3 feature coverage and completion format? Either would help clarify what the SFT gains demonstrate.
-
What tools can GPT-5.6-Terra-Agent actually use? Can it execute programs and revise them using compiler or simulator feedback, and what are its iteration and inference budgets? The paper also cites QUASAR, which releases a model for tool-augmented OpenQASM generation. Is that checkpoint suitable for evaluation on QASM-Eval? Its training tasks differ from those here, but testing it could provide a useful transfer baseline.
Thank you for identifying these counterexamples! We examined the reported cases and updated the verifier, manuscript, and GitHub repository. Re-scoring affects ~5% of tasks results. We report the revised scores below and have updated the paper accordingly.
Q1. The three counterexamples and the fix
The released verifier matches the original evaluation implementation. We have also updated the released code based on your careful audit.
The three cases require the following decisions:
timing/theme01/v1/instance1(wrong RX angle). Timing tasks were designed to test scheduling, so the original rule checked only the timeline. Because every RX gate has the same duration regardless of angle, a wrong angle kept the schedule intact and passed. This answer should fail, since it does not follow the TODO prompt. In the updated version it requires both the distribution and the timeline to match.classical/theme17/v1/instance2(zero RZ angles). This answer should also fail. Our templates were built around output distributions because it is the observable result of most quantum programs. A phase-comparison routine existed in the code but was not part of the acceptance rule. The updated version now makes the phase-sensitive state check mandatory for all categories, and this answer is now rejected.timing/theme09/v1/instance1(durationof).durationof({ x q[0]; })should evaluate to 40 ns under the device configuration. The original implementation incorrectly returned a fallback of onedt(1 ns) without evaluating the enclosed operation. We corrected this error: the updated version evaluates the block on an isolated scheduler copy using the configured duration of each gate.Q2. Effect on scores and rankings
Re-scoring affects ~5% of tasks results. Following the updates prompted by your counterexamples, the main results are shown below and have been incorporated into the revised manuscript.
The ranking of all settings is unchanged.
We also selected 100 tasks, with 25 per category, and injected parameter, phase, and duration errors.
In this controlled test, the updated verifier rejects all incorrect variants, yielding 0.00% false acceptance and 3.00% false rejection.
Q3. Independently test set
We also evaluated tasks originating outside the generation pipeline. We assembled 50 tasks from official OpenQASM examples and documentation and Qiskit circuit-library instances, covering all four categories. The TODO prompts for these tasks were generated by GPT-6.1-sol and manually reviewed by us, and the tasks were selected after examining these generated prompts. The table reports mean pass@1 over three samples per task.
The SFT(ours) models still hold the advantage.
Q4. GPT-5.6-Terra-Agent and QUASAR
The GPT-Terra retrieval setup combines a BM25 sparse retriever and a bge-m3 dense retriever and supplies the retrieved examples. GPT-Terra can decide whether to retrieve and inspect the results. This happens in one run without iterative execution or repair.
This is intentional. Our aim is to identify gaps in the model's own knowledge of OpenQASM, then address the gaps through targeted training. Interactive and tool-assisted generation is our next direction and calls for more complex, project-level benchmarks, analogous to the progression from function-level HumanEval to project-level SWE-Bench. We discuss this direction in the revised paper's future-work section.
We also evaluated the public QUASAR checkpoint,
Benyucong/rl_quantum_4b, on the common 100-task subset, where Astra, Fable, and Carbon score 73%, 78%, and 75%. QUASAR scores 0/100. This not surprising because its training focus on specialized tasks such as QAOA and VQE. whereas our tasks require more general knowledge.