acceptodds
Under review as a conference paper at ICLR 2027

QASM-Eval: A Dataset to Train and Evaluate LLMs on OpenQASM-3 Beyond Quantum Circuits

Abstract

Quantum computing remains in the Noisy Intermediate-Scale Quantum (NISQ) era, with performance constrained by noise. Addressing this limitation requires hardware-facing capabilities beyond gate sequences: mid-circuit measurement and classical feedback for quantum error correction (QEC), precise timing for dynamical decoupling (DD), and pulse-level waveform access for calibration. OpenQASM 3 exposes these capabilities through a hardware-level programming interface. Despite rapid progress in large language models (LLMs) for code generation, datasets targeting these advanced features remain lacking. We introduce QASM-Eval, the first comprehensive dataset to train and evaluate LLMs on OpenQASM 3, targeting completion of hardware-facing constructs within a supplied program context. QASM-Eval comprises an expert-curated test set of 1,200 tasks and a training set of over 4,000 tasks, covering classical logic, timing scheduling, pulse control, and complex tasks inspired by quantum-control workflows. An extended verifier automatically checks syntax, quantum measurement distributions, and program timelines. Our evaluation identifies substantial headroom for OpenQASM 3 code generation and significant gains from targeted fine-tuning. Fine-tuned Llama-3-8B outperforms zero-shot GPT-5.6-Terra, while fine-tuned Llama-3-70B achieves 61.17% overall pass@1 and approaches few-shot-augmented GPT-5.6-Terra. QASM-Eval provides a benchmark and training resource for specification-guided code generation over the hardware-facing constructs of OpenQASM 3, supporting the development of LLM assistants for quantum programming. Data and code: https://anonymous.4open.science/r/QASM-Eval-0400/

Primary area:
datasets and benchmarks
Keywords:
Large language models, code generation, quantum programming, OpenQASM 3, benchmarks and datasets, execution-based evaluation, supervised fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VhcNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2(−1)
Soundness
2(+1)
Read review
Paper summary

This paper introduces QASM-Eval, a training dataset and executable benchmark for OpenQASM 3 code completion beyond conventional gate-level circuits. It covers classical control, timing, pulse operations, and combinations of these capabilities through 1,200 test tasks across 100 themes, together with approximately 4,000 training examples. An extended verifier checks syntax, required constructs, measurement distributions, and execution timelines. The experimental study evaluates zero-shot generation, cross-theme few-shot prompting, a documentation-augmented agent, and LoRA supervised fine-tuning using QASM-Eval and existing quantum-programming datasets. The principal contribution is the task coverage, dataset, and evaluation infrastructure.

Critical strengths
  1. The focus on OpenQASM 3 features beyond gate sequences is useful. Classical feedback, timing constraints, and pulse control introduce requirements that conventional circuit-generation tasks do not adequately test. Giving these features dedicated tasks, including tasks that combine them, is a sensible contribution to OpenQASM 3 program generation.

  2. The experiments include useful controls. The authors evaluate several model families and compare fine-tuning against both same-model few-shot prompting and training on existing datasets. The theme-held-out split is also preferable to randomly separating variants of the same theme. For Llama-8B, QASM-Eval fine-tuning raises pass@1 from 32.87% with three-shot prompting to 46.93%, providing evidence that the training data helps on the covered tasks.

Critical weaknesses
  1. The verifier accepts some incorrect completions and rejects a valid timing alternative. Checks of the unmodified released code produced the following results. These checks used the default 8,000-shot setting, and the reference completions passed.
  • In timing/theme01/v1/instance1, changing the requested rx(1.5708) to rx(0) still passes. The measurement-distribution check detects the difference, but the final decision ignores it because the timeline matches.
  • In classical/theme17/v1/instance2, setting the RZ angles to zero while keeping the loop and arithmetic also passes. The resulting state has fidelity approximately 0.418 with the reference state. Their computational-basis measurement distributions nevertheless coincide, so the test misses the phase error.
  • In timing/theme09/v1/instance1, the requested delay is the duration of an X gate. The released default device configuration assigns 40 ns to X, but durationof({ x q[0]; }) evaluates to 1 ns. A fixed 1 ns delay is therefore accepted, while the corresponding 40 ns delay is rejected. These failures affect what a passing score means. In the first two cases, the completion does not perform the operations specified in the prompt. The third case introduces an inconsistency between the interpreter’s duration calculation and its own device configuration. The reported 94.5% expert agreement on 200 outputs does not resolve these problems. The checks above are targeted counterexamples, so they do not establish how often such errors occur or whether model rankings change. That impact needs to be measured before the main results can be trusted as an assessment of program correctness.
  1. It remains unclear how much of the SFT improvement transfers to independently written tasks. The theme-held-out split is a useful precaution, but both sides still share the task-generation process and description style. Many prompts give detailed instructions for a short completion inside a supplied program. The experiments do not show how well the trained models handle specifications written independently of that process. The comparisons with QCircuitBench and Agent-Q leave a related uncertainty. Matching training token counts does not match OpenQASM 3 feature coverage or completion format. Their performance gap therefore does not isolate the benefit of the proposed construction procedure. The same-model few-shot results give a more informative comparison here. For Llama-70B, overall pass@1 increases from 55.28% to 61.17% after fine-tuning, but on Complex tasks the change is only 33.07% to 33.93%. The evidence for an SFT advantage on those combined workflows is modest. An independently authored test subset, or a closer training-data control, would help determine which gains carry beyond the benchmark’s particular format.
Questions for authors
  1. Does the released verifier match the version used for the reported experiments? Please explain the expected decisions for the three cases above, especially the handling of incorrect gate parameters in Timing tasks and the implementation of durationof.

  2. How do the scores change after these issues are corrected? Re-scoring saved model outputs would help establish their impact. Tests containing deliberately incorrect programs and valid equivalent alternatives would also be useful for checking whether the same problems occur elsewhere in the four categories.

  3. Do you have results on tasks or descriptions written independently of the generation pipeline? If not, is there a training comparison that more closely matches the OpenQASM 3 feature coverage and completion format? Either would help clarify what the SFT gains demonstrate.

  4. What tools can GPT-5.6-Terra-Agent actually use? Can it execute programs and revise them using compiler or simulator feedback, and what are its iteration and inference budgets? The paper also cites QUASAR, which releases a model for tool-augmented OpenQASM generation. Is that checkpoint suitable for evaluation on QASM-Eval? Its training tasks differ from those here, but testing it could provide a useful transfer baseline.

1 comment

Thank you for identifying these counterexamples! We examined the reported cases and updated the verifier, manuscript, and GitHub repository. Re-scoring affects ~5% of tasks results. We report the revised scores below and have updated the paper accordingly.

Q1. The three counterexamples and the fix

The released verifier matches the original evaluation implementation. We have also updated the released code based on your careful audit.

The three cases require the following decisions:

  • timing/theme01/v1/instance1 (wrong RX angle). Timing tasks were designed to test scheduling, so the original rule checked only the timeline. Because every RX gate has the same duration regardless of angle, a wrong angle kept the schedule intact and passed. This answer should fail, since it does not follow the TODO prompt. In the updated version it requires both the distribution and the timeline to match.
  • classical/theme17/v1/instance2 (zero RZ angles). This answer should also fail. Our templates were built around output distributions because it is the observable result of most quantum programs. A phase-comparison routine existed in the code but was not part of the acceptance rule. The updated version now makes the phase-sensitive state check mandatory for all categories, and this answer is now rejected.
  • timing/theme09/v1/instance1 (durationof). durationof({ x q[0]; }) should evaluate to 40 ns under the device configuration. The original implementation incorrectly returned a fallback of one dt (1 ns) without evaluating the enclosed operation. We corrected this error: the updated version evaluates the block on an isolated scheduler copy using the configured duration of each gate.

Q2. Effect on scores and rankings

Re-scoring affects ~5% of tasks results. Following the updates prompted by your counterexamples, the main results are shown below and have been incorporated into the revised manuscript.

Model / settingOriginal pass@1Updated pass@1
Llama-8B-base10.77%7.43%
Llama-8B-fs37.20%34.95%
Llama-8B-ours51.82%49.15%
Llama-70B-base17.37%16.08%
Llama-70B-fs62.43%60.67%
Llama-70B-ours64.08%63.42%
GPT-5.6-Terra43.18%40.90%
GPT-5.6-Terra-fs70.88%69.03%
GPT-5.6-Terra-Agent52.90%51.27%
DPSK-V4-Pro28.60%26.45%
DPSK-V4-Pro-fs69.38%67.78%

The ranking of all settings is unchanged.

We also selected 100 tasks, with 25 per category, and injected parameter, phase, and duration errors.

CategoryFalse acceptFalse reject
Classical0.00%4.00%
Timing0.00%0.00%
Pulse0.00%8.00%
Complex0.00%0.00%
Overall0.00%3.00%

In this controlled test, the updated verifier rejects all incorrect variants, yielding 0.00% false acceptance and 3.00% false rejection.

Q3. Independently test set

We also evaluated tasks originating outside the generation pipeline. We assembled 50 tasks from official OpenQASM examples and documentation and Qiskit circuit-library instances, covering all four categories. The TODO prompts for these tasks were generated by GPT-6.1-sol and manually reviewed by us, and the tasks were selected after examining these generated prompts. The table reports mean pass@1 over three samples per task.

ModelBase3-shot(fs)SFT(ours)
Llama-8B21.3%22.67%26.0%
Llama-70B46.67%52.00%58.00%

The SFT(ours) models still hold the advantage.

Q4. GPT-5.6-Terra-Agent and QUASAR

The GPT-Terra retrieval setup combines a BM25 sparse retriever and a bge-m3 dense retriever and supplies the retrieved examples. GPT-Terra can decide whether to retrieve and inspect the results. This happens in one run without iterative execution or repair.

This is intentional. Our aim is to identify gaps in the model's own knowledge of OpenQASM, then address the gaps through targeted training. Interactive and tool-assisted generation is our next direction and calls for more complex, project-level benchmarks, analogous to the progression from function-level HumanEval to project-level SWE-Bench. We discuss this direction in the revised paper's future-work section.

We also evaluated the public QUASAR checkpoint, Benyucong/rl_quantum_4b, on the common 100-task subset, where Astra, Fable, and Carbon score 73%, 78%, and 75%. QUASAR scores 0/100. This not surprising because its training focus on specialized tasks such as QAOA and VQE. whereas our tasks require more general knowledge.

Reviewer_yJR8OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces QASM-Eval, a dataset for training and evaluating LLMs on OpenQASM 3, focusing on classical control, timing, pulse control, and complex quantum-control tasks. It includes 1,200 test tasks and over 4,000 training tasks, with an automated verifier for syntax, measurement distributions, and execution schedules. Fine-tuning improves Llama-8B from 9.98% to 46.93% pass@1 and Llama-70B from 15.70% to 61.17%.

Critical strengths
  • The paper addresses a gap in existing quantum-code benchmarks.
  • The dataset covers different aspects of quantum programming, including classical feedback, timing constraints, pulse control, and their combinations.
  • The evaluation goes beyond syntax checking by considering quantum measurement distributions and execution timelines.
  • The experiments include different model sizes, fine-tuning baselines, few-shot prompting, and theme-held-out evaluation, with performance improvements.
Critical weaknesses
  1. QCircuitBench already uses OpenQASM 3 for circuit representation, while Agent-Q addresses quantum circuit generation. The extension to classical control, timing, and pulse operations distinguishes this paper. However, how much of the proposed verifier is new compared to existing OpenQASM/OpenPulse tools? A more detailed comparison of the supported features and verification capabilities would strengthen the novelty claim.
  2. Both training and test tasks come from the same template-generation pipeline, although the authors exclude test themes from training. This tests generalization across themes but not necessarily to programs written independently of these templates. Can the authors evaluate the models on tasks written by quantum programmers without using the generation pipeline?
  3. The authors validate the verifier in Appendix E, but comparing final measurement distributions does not guarantee that two programs implement the same quantum operation. For example, different states can produce identical measurement distributions in the computational basis. Could this lead to incorrect programs being accepted? Can the authors quantify false acceptance rates using additional input states, measurement bases, or independent verification?
  4. The paper motivates the benchmark using QEC, dynamical decoupling, and calibration, but the evaluation uses simulated quantum states and a virtual scheduling model. It is unclear whether programs passing the verifier would also satisfy hardware-specific constraints. Could the authors test a subset of generated programs using an actual hardware backend or provider-specific compiler?
  5. Although the overall improvements are substantial, Llama-70B achieves only 33.93% pass@1 on complex tasks, compared with 85.3% on timing and 89.3% on pulse tasks. Since combining these features is one of the main motivations, this gap deserves further analysis. Which combinations of features cause the most failures?
  6. The authors compare against models fine-tuned on QCircuitBench and Agent-Q using matched token budgets. However, these datasets do not target the same advanced OpenQASM 3 features, making the performance gap somewhat expected. Could the authors include ablations on individual task categories or template variations to identify which parts of QASM-Eval training contribute most to the improvements?
Questions for authors
  1. Both training and test tasks are generated using the same template-based pipeline, although test themes are held out. How can we determine whether the models learn OpenQASM 3 programming rather than recurring template structures? Is there any evaluation on independently written tasks?
  2. The verifier compares final measurement distributions, but different quantum states can produce identical measurement outcomes in the computational basis. How are such cases handled? Could this result in incorrect programs being accepted, particularly for pulse-control and complex tasks?
  3. The evaluation relies on simulated quantum states and virtual execution schedules. How are hardware-specific constraints, such as channel conflicts, pulse compatibility, and controller latency, handled? Is there proof that programs passing the verifier would also compile on an actual quantum backend?
  4. Llama-70B achieves only 33.9% pass@1 on complex tasks, compared with 85.3% on timing and 89.3% on pulse tasks. Which feature combinations contribute most to these failures? Is the performance drop mainly due to scheduling, quantum-state errors, or interactions between different features?
  5. QCircuitBench and Agent-Q focus on different programming features from QASM-Eval. To what extent do the reported improvements come from learning advanced OpenQASM 3 constructs rather than adapting to the dataset’s task format? Could category-specific ablations help distinguish these effects?
5 comments

Thank you for your detailed questions! Most of them concern generalization, verification, and hardware assumptions, which we also considered closely during this work. Below, we add an independent task set from public Github sources, a fault-injection test of the verifier, and a breakdown of Complex failures. The revised appendix also includes which OpenQASM 3 features the verifier supports.

Q1. Generalization beyond template & independent test set

We assembled 50 additional tasks from official OpenQASM repository (https://github.com/openqasm/openqasm) and Qiskit circuit-library (https://github.com/Qiskit/qiskit), covering all four categories. The results on Llama family are below:

ModelBase3-shot(fs)SFT(ours)
Llama-8B21.3%22.67%26.0%
Llama-70B46.67%52.00%58.00%

These cases show the SFT(ours) models applying learned constructs beyond our template structures. Public OpenQASM training material remains scarce, and current LLMs have considerable room to improve (Even GPT-6 Astra, Claude Fable 5.2, and Gemini 4 Carbon can only achieve 73%, 78%, and 75% pass@1 on the 100-task subset). Training on the language's syntax and its connection to quantum semantics provides a direct way to address this gap.

Q2. Identical distributions problem

We used several approaches to prevent this. First, we implemented a phase-sensitive comparison interface alongside the distribution check. Second, we use randomized background to diversify the initial quantum states. Third, experts specify operations on qubits in each core-task template, minimizing ambiguity.

To assess these approaches, we constructed a subset of 100 tasks and injected parameter, phase, and duration errors (that may cause identical distributions problem). The updated verifier gives the following results:

CategoryFalse acceptFalse reject
Classical0.00%4.00%
Timing0.00%0.00%
Pulse0.00%8.00%
Complex0.00%0.00%
Overall0.00%3.00%

For Pulse tasks, we conducted an additional comparison: 30 pulse configurations were injected with minor changes in waveform. The false accept rate is still 0%.

For Complex tasks, the updated verifier gives 0.00% false acceptance and false rejection, which shows that the new verifier now works as intended.

Q3. Hardware constraints

We selected 20 answers that passed the verifier and successfully compiled them on a collaborator's physical quantum controller. This was a preliminary check and was not included in the original manuscript.

We would like to clarify that OpenQASM 3 is similar to RISC-V in this respect: the language usually defines more features than any single device implements, and each control stack supports its own subset (e.g., QubiC, https://arxiv.org/abs/2404.15260). Our verifier therefore exposes general interfaces for the constraints that are common across backends, such as ports, gate durations, and controller latency. Channel conflicts and pulse compatibility depend on the specific backend implementation. The simulator provides extension points for them, but modeling every vendor's hardware is outside this benchmark's scope. We have added this discussion to the limitations section.

Q4. feature combinations that cause Complex failures

The failures concentrate where measurement-dependent control must be coordinated with timing. For Llama-70B SFT (ours), answers combining classical control, timing, and pulses are rejected 62.67% of the time, and the timeline check is the most frequent cause of rejection. Details can be found below:

Theme groupLlama-70b-ours rejection
Measurement-driven branching and feedback correction52.5%
Conditional pulses aligned to the end of a time window91.7%
Repeated pulses with phase alternation (dynamical decoupling)86.7%
Raw waveform acquisition and filtering30.0%

Q5. Advanced OpenQASM 3 constructs and task-format adaptation

QCircuitBench and Agent-Q do not cover the advanced OpenQASM 3 constructs and therefore perform poorly on our benchmark. The table below gives the category-level breakdown of the Llama improvements in Figure 2.

Category8B-fs8B-ours70B-fs70B-ours
Classical+39.40%+42.34%+36.00%+28.40%
Timing+38.94%+41.74%+52.20%+57.86%
Pulse+27.93%+68.73%+63.07%+66.07%
Complex+3.80%+14.07%+27.07%+37.00%

SFT training improve performance the most across all four categories. Figure 5(a) also shows semantic recovery: tasks where the base model produced syntactically valid but semantically incorrect code are solved correctly after SFT training. The recovery rates are 48.2% for Llama-8B-ours and 50.5% for Llama-70B-ours.

Reviewer_yJR8OpenReview ↗

comment

Thank you for the responses and additional experiments. My main remaining concern is the verifier's semantic correctness. The zero false acceptance rate is a positive result, but the tests cover only selected error types. Could you clarify what the phase-sensitive comparison verifies and explain the 8% false rejection rate for Pulse tasks?

Regarding hardware validation, it remains unclear whether the 20 compiled programs were executed and whether hardware-specific constraints were verified. Also, the category-level results do not fully distinguish learning advanced OpenQASM 3 features from adapting to benchmark templates.

Thank you for your continued engagement and the follow-up questions! We address your remaining questions below.

The selected error types

The error types we chose are the ones you pointed out earlier, where different code produces quantum states with the same measurement distribution (Identical distributions problem). (1) Phase errors: changing the relative phase of a quantum state does not change the distribution. (2) Duration errors: on short timescales, where decoherence is not significant, the length of a delay does not change the distribution either. Other routes to an identical distribution, such as a different complex sequence of state transformations, will deviate from what the TODO prompt asks for, and fail the existing element check.

Phase-sensitive check

Phase is the main source of the problem you raised, where different code yields identical distributions. Some quantum gates exist specifically to change phase, so we added a phase-sensitive check. The other related error type (duration) is already covered by the schedule check. For each task, we compute the exact pre-readout quantum state density matrices of the reference and the candidate, enumerating mid-circuit measurement and reset branches. We then compare their trace distance. A candidate passes only if the trace distance is ≤ 0.05, the same threshold we use for the total variation distance in Appendix D.

The 8% false rejection of Pulse tasks

These false rejections occurred only on the test set we built specifically to examine the identical-distribution problem you raised. We confirmed that these false rejections never appear in our actual experimental data, so they do not affect any result in the main text. The false rejection happens when randomly changing an extern declaration, such as float[32] to float[(32+0)] (This looks intuitive to humans but is quite unconventional in OpenQASM). They are purely a classical parsing issues, unrelated to quantum pulse semantics. This is because when building QASM-Eval, we focused on this kind of classical operations in the program logic rather than on declarations, so our overly strict parser rejected this harmless form if it appears in certain code sections. After we added support for this syntax, the false-rejection rate became 0%. We also re-checked all results, and the parsing change did not alter any number reported in the main text.

Hardware

Yes, our collaborator compiled all 20 programs and ran them on their superconducting chip. Existing lab-level quantum computers each support only their own chosen subset of OpenQASM 3 hardware features. What the run on our collaborator’s chip validated is therefore part of the hardware constraints (delay, duration, port, and frame interfaces). The other hardware-related features such as set_phase or set_frequency were not tested on that machine, but we implement all of them strictly according to the official OpenQASM 3.0 specification. Therefore, a backend that supports these features can load its own backend constraints, such as the control delay you mentioned or pulse rescheduling to resolve channel conflicts, and then compile the program into pulse sequences and execute it.

Adapting to benchmark templates

We agree that a category-level table alone is not sufficient. The table was meant to compare SFT against the 3-shot baseline, which also sees the QASM-Eval task format, so the gap between the two reflects a gain from SFT training.

Beyond this, we provide evidence from two new directions that the model has actually learned advanced constructs.

First, in our first-round response (and to Reviewers Heni and VhcN), we assembled 50 independent tasks from the official OpenQASM repository (https://github.com/openqasm/openqasm) and the Qiskit circuit library (https://github.com/Qiskit/qiskit). The code for these tasks does not come from our pipeline's templates, and the TODO prompts were generated by a different LLM (gpt-6.1-sol) from the one used in our pipeline (Qwen3-Coder-480B). The table below reports the results:

ModelBase3-shot(fs)SFT(ours)
Llama-8B21.3%22.67%26.0%
Llama-70B46.67%52.00%58.00%

On these tasks, SFT (ours) outperforms both 3-shot prompting (fs) and the base model. This shows that the SFT models can apply the constructs they learned to programs outside the benchmark generator.

Second, Figure 5(a) in the main paper reports the semantic recovery that SFT training produces. Errors that are purely semantic (with no syntax error involved) are corrected after training. On these tasks, the semantic recovery rate after fine-tuning was 48.2% for Llama-8B-ours and 50.5% for Llama-70B-ours. This suggests that the model has learned how OpenQASM features relate to quantum behavior, rather than simply following templates.

Reviewer_HeniOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

QASM-Eval is an interesting benchmark paper that moves beyond gate-level representations and optimizations for quantum circuits -- this is very much needed and helpful. I have personally worked on code generation for quantum circuits and one of the dissatisfying scenarios has been the lack of eval frameworks that also include pulse level control for manipulating internal states of quantum systems and classical readouts of intermediate computation results for error correction.

Critical strengths

Strengths:

  1. The paper is addressing a real gap that I and others have experienced in this area.
  2. The data and code have been linked and the task is itself not too computationally expensive.
Critical weaknesses

Weaknesses:

  1. Observations about fine-tuning gains are not surprising; there are many tasks where fine-tuning does well often at cost of poor performance on other tasks.
  2. While experts have reviewed the data, the data it itself LLM generated. This is a major concern for me, in terms of how useful this data set could be. Can we at least have an annotation for each task saying what is it exactly that the humans verified for that tasks and how many experts at what level of expertise performed that validation.
Questions for authors

Why compare with the GPT Terra model? Can we at least take a challenging core set and compare with SOTA models?

Can you explain the nature of human effort put in this data generation and curation per data element? For every data item, did a human being read it, run it, study it, curate it? How many people did this? What are their levels of expertise?

Why not release your benchmark generator and let others jus generate this benchmark from SOTA models?

1 comment

Thank you for your review! We have added results for three frontier models (Astra, Fable, and Carbon) on a 100-task subset, together with 50 tasks derived from public GitHub repositories. We would also like to clarify that our release includes the benchmark generator in dataset_factory/. This directory contains the task-generation scripts and allows researchers to add their own tasks, generate custom test sets, and evaluate other LLMs using the evaluation scripts.

Q1. Comparison with SOTA models & independent test set

GPT-5.6 was the latest model when we conducted the original experiments, before the September release of GPT-6. Following your suggestion, we additionally evaluated GPT-6 Astra, Claude Fable 5.2, and Gemini 4 Carbon on the same 100 tasks, with 25 tasks per category and thinking-level set as 'high'. The table reports pass@1.

ModelClassicalTimingPulseComplexAverage
GPT-6 Astra72.0%96.0%80.0%44.0%73.0%
Claude Fable 5.276.0%100.0%88.0%48.0%78.0%
Gemini 4 Carbon76.0%100.0%84.0%40.0%75.0%

Even these frontier models leave 22–27% of the common subset unsolved. As discussed in the paper, publicly available OpenQASM code is limited, leaving models with relatively little training material for this language. Their performance still has substantial room for improvement and this gap is what motivated this benchmark.

To examine performance on programs originating outside our generator and templates, we assembled 50 independent tasks from official OpenQASM repository (https://github.com/openqasm/openqasm) and Qiskit circuit-library (https://github.com/Qiskit/qiskit). The table below reports the results:

ModelBase3-shot(fs)SFT(ours)
Llama-8B21.3%22.67%26.0%
Llama-70B46.67%52.00%58.00%

On these tasks, SFT(ours) exceeds 3-shot prompting (fs) and base model performance. It show that the SFT(ours) models can apply learned constructs to programs outside the benchmark generator. Teaching OpenQASM syntax together with the connection between code and quantum behavior can therefore support generalization.

Q2. Human effort and expertise

The experts involved in dataset construction have several years of experience using quantum computers (superconducting). Their experience includes work supported by the IBM Quantum Researchers and Educators Program and collaborations involving laboratory quantum platforms that use a subset of OpenQASM.

Their responsibilities covered the following stages:

  1. They manually wrote the background quantum programs to provide diverse initial quantum states.
  2. They manually wrote the task template for each theme: a relevant OpenQASM program, together with the variables, ports, qubits, and parameters that could be randomized.
  3. LLMs generated three additional variants for each theme, such as different branching structures for an if/else task. Experts reviewed the variants across all 100 themes for consistency with the theme and for syntactic and semantic correctness.
  4. Experts sampled the LLM-generated TODO descriptions for 200 test tasks and 400 training tasks. They checked that the descriptions matched the OpenQASM code and did not disclose the answers.
  5. We also selected 200 test tasks and generated answers with GPT-Terra. One expert and the verifier scored these answers independently, reaching 94.5% agreement.

Q3. Releasing the generator

The release package includes dataset_factory/, with background and core-task generators for all four categories, prompt-generation code, and dataset assembly utilities. The entry point is dataset_factory/generate_dataset.py. Researchers can generate the full dataset, create subsets of any size, or add new tasks of their own.

We have highlighted this entry point in Reproducibility Statement section the revised paper

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Revisions

We thank all reviewers for their constructive feedback. As the discussion period ends soon, we briefly summarize the main changes for convenience. The revised PDF has been uploaded.

  1. Verifier correctness (Reviewers VhcN, yJR8). We fixed the three counterexamples reported by Reviewer VhcN. Re-scoring changed ~5% of task results, and the ranking of all settings is unchanged. A fault-injection test on 100 tasks (parameter, phase, and duration errors) gives 0% false acceptance and 3% false rejection.

  2. Frontier models (Reviewers Heni, VhcN). We evaluated GPT-6 Astra, Claude Fable 5.2, and Gemini 4 Carbon on a 100-task subset (73–78% pass@1), and the QUASAR checkpoint (0/100).

  3. Tasks outside the generator (Reviewers yJR8, VhcN, Heni). We added 50 tasks derived from the official OpenQASM repository and the Qiskit circuit library. SFT models outperform both base and 3-shot settings for Llama-8B and Llama-70B.

  4. Analysis and clarifications (Reviewers Heni, yJR8, VhcN). We added a breakdown of Complex-task failures, category-level gains, details of expert involvement in dataset construction, the tool setup for the GPT-Terra agent, and a discussion of hardware constraints, including a preliminary compilation check of 20 passing programs on a physical controller.

  5. Release (Reviewer Heni). The benchmark generator (dataset_factory/generate_dataset.py) is included in the release and highlighted in the appendix.

We would be grateful if the reviewers could let us know whether these responses address their concerns, and we are happy to provide further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.