- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses semantic correctness in LLM-based scientific code generation by focusing on Clifford circuit synthesis for quantum state preparation. The authors introduce a target-conditioned framework where targets are specified as compact signed stabilizer generators, verified using exact classical verifiers. The models are supervised using Aaronson-Gottesman chain-of-thought (AG-CoT) algorithmic traces and further trained on verifier-accepted completions. Controlled experiments across 3B, 7B, and 32B model families demonstrate substantial gains in state-equivalence accuracy compared to circuit-only baselines.
Critical strengths
- Supervising circuit generation with verified Aaronson-Gottesman reduction traces significantly improves exact quantum state equivalence over direct circuit-only fine-tuning across multiple model scales. 2. Formulating target states using compact signed stabilizer generators instead of full state vectors dramatically reduces token context length while maintaining exact classical verifiability.
Critical weaknesses
- Limitations of prior quantum LLM benchmarks and fine-tuning techniques are not sufficiently analyzed to show why general-purpose CoT or standard execution-based verification fail without domain-specific algorithmic trace alignment. 2. The overall methodology lacks a high-level algorithmic description or flowchart detailing the full data construction, inverse compilation, and verifier filtering loops to cleanly decouple novel contributions from existing components like Aaronson-Gottesman reduction. 3. Results in tables and text are presented too concisely without bulleted observational points, leaving key metrics like the gap between strict label matching and state equivalence under-discussed. 4. All empirical evaluations are restricted solely to Clifford random-circuit benchmarks derived from QCircuitBench, lacking evaluation on broader non-Clifford or realistic physical quantum error correction circuits.
Questions for authors
- In Section 3, tableau semantics are introduced, but it is not explained why existing code generation fine-tuning strategies cannot learn this representation directly. Can the paper clarify the domain limitations of existing approaches? 2. In Section 4, could the work provide a complete step-by-step top-level algorithm or pseudocode for the AG-CoT trace generation and inverse compilation pipeline, explicitly marking which modules use existing Qiskit routines versus novel verifier logic? 3. In Section 5, what are the exact compute requirements and tool flows (e.g., vLLM or TRL configurations) used during the RFT generation phase? 4. In Section 6.2 (Table 3 and Table 4), the performance gap between strict saved-label matching and full state equivalence is significant. Can the paper provide detailed observational points analyzing why label representation misalignment occurs so frequently despite valid execution? 5. In Section 6, evaluation is restricted to random Clifford targets. How does the AG-CoT approach perform when applied to structured quantum error correction benchmarks such as StabilizerBench?
4 comments
Response to Reviewer bfjw (2/3)
W2 + Q2. Construction pipeline and reused components
Reviewer concern. The overall methodology lacks a high-level algorithmic description or flowchart detailing the full data construction, inverse compilation, and verifier filtering loops to cleanly decouple novel contributions from existing components like Aaronson-Gottesman reduction.
In Section 4, could the work provide a complete step-by-step top-level algorithm or pseudocode for the AG-CoT trace generation and inverse compilation pipeline, explicitly marking which modules use existing Qiskit routines versus novel verifier logic?
Response.
The full procedure is given below as one algorithm, and §4 now gives it in condensed form as Algorithm 1 (p6), which merges steps 1–2 and 6–7. [Qiskit] marks an existing Qiskit routine, [AG] the published Aaronson–Gottesman reduction, and [ours] logic written for this work. Figure 2 (p5) shows this pipeline as a flowchart, and Figures 6–7 in Appendix C (pp17–18) show the 32B variant.
Teacher construction for a target given by n signed generators (Qwen-3B and Mistral-7B)
- [Qiskit] Build the target state (
StabilizerState.from_stabilizer_list). - [ours] Row-reduce the generators to a canonical basis.
- [Qiskit] Complete the canonical basis to a full Clifford tableau, which adds n destabilizer rows.
- [AG] Reduce the tableau to the identity, one qubit at a time, with h, s, cx and swap operations and final sign corrections. Each tableau update is a Qiskit routine.
- [ours] After every operation, write the operation and the current destabilizer and stabilizer rows of that qubit into the trace.
- [Qiskit] Invert the recorded reduction to obtain the preparation circuit.
- [ours] Expand every gate into {h, s, cx}. [Qiskit] Serialize the circuit to OpenQASM 3.
- [ours] Accept the pair only if the serialized program parses, uses only {h, s, cx}, has n qubits, and prepares the target state. Parsing and the state comparison (
StabilizerState.equiv) are Qiskit routines. - [ours] Emit two completions for the same prompt, trace plus program for AG-CoT, and the program alone for Direct SFT.
Verifier-filtered continuation
- [vLLM] Sample one completion per training-pool prompt from the AG-CoT model.
- [ours] Keep a completion only if generation ended normally, it contains one trace block followed by one fenced program, and the program starts with
OPENQASM 3.0;and passes the checks of step 8. - [Transformers, PEFT] Continue training from the AG-CoT adapter on the kept completions, unchanged.
In the 32B study, steps 1–3 start from the Clifford of the QCircuitBench reference circuit, and step 8 requires saved-label equality (§4, p4).
The AG reduction and the Qiskit state routines are existing components. The contribution is what is built around them, a teacher derived from the target generators alone, the recorded reduction used as supervision against a circuit-only control with the same prompt and program, and one exact state check shared by teacher data, continuation data, and evaluation.
Q3. Generation tools and compute requirements
Reviewer concern. In Section 5, what are the exact compute requirements and tool flows (e.g., vLLM or TRL configurations) used during the RFT generation phase?
Response.
The continuation phase has three stages. The table gives the tool flow and the logged compute of the Mistral-7B run, on eight NVIDIA RTX A6000 GPUs.
| Stage | Tool flow | Compute |
|---|---|---|
| Proposal generation | vLLM 0.6.4.post1 with the AG-CoT LoRA adapter, BF16. 8,192 training-pool prompts in 8 shards of 1,024, one completion per prompt, batches of 16, temperature 0.8, top-p 0.95, seed 42, 8,192-token context with up to 8,000 new tokens | One GPU per shard, 3 h 23 min to 3 h 36 min per shard |
| Verification | Qiskit 2.3.0 with the acceptance rule of §4 (p6), no repair. 983 completions accepted | 8 CPU processes, 25 min including the shard merge |
| Continuation training | Transformers 4.52.4 Trainer with PEFT 0.19.1 on PyTorch 2.5.1, starting from the AG-CoT adapter. One epoch, learning rate 5e-6, global batch 32, BF16 | 8 GPUs, 31 updates in 4 min |
The Qwen2.5-3B continuation uses the same sampler, the same sampling parameters, and the same verifier. It accepts 715 of 8,192 completions and trains for 23 updates on one GPU with the same learning rate and global batch size. TRL is used only for GRPO in the 32B study. Table 11 (p21) lists its version.
This table is now Table 28 in Appendix J.3 (p29).
Response to Reviewer bfjw (3/3)
W3 + Q4. Label matching, state equivalence, and presentation of results
Reviewer concern. Results in tables and text are presented too concisely without bulleted observational points, leaving key metrics like the gap between strict label matching and state equivalence under-discussed.
In Section 6.2 (Table 3 and Table 4), the performance gap between strict saved-label matching and full state equivalence is significant. Can the paper provide detailed observational points analyzing why label representation misalignment occurs so frequently despite valid execution?
Response.
The observations are listed below, and §6.2 now lists observations 1, 3 and 4 under Tables 3 and 4 (p9).
- Both checks start from a valid circuit. Both are applied only after the program parses and converts to a Clifford circuit, which holds for 99.6–99.9% of trained-model outputs (Table 3, p9). The gap between the two checks is therefore not an execution failure.
- Label matching compares one generator list, and state equivalence compares the state. Executing a circuit from the all-zero state gives one list of n signed generators. Label matching accepts the output only if this list equals the saved list, which is the one produced by the reference circuit, up to order. State equivalence, computed with the Qiskit routine
StabilizerState.equiv, accepts it if the two lists describe the same state (§3, pp3–4). - One state has many generator lists. The two-qubit program
h q[0]; cx q[0], q[1];yields {+XX, +ZZ}. Placingcx q[0], q[1];in front of it prepares the same state and yields {+XX, −YY}. A model that reaches the target state through a gate sequence different from the reference can therefore end on a different list and fail label matching. - About half of the correct states are affected. In Table 3, label matching and state equivalence give 2.62% and 5.37% for Basic AG-CoT, 2.31% and 4.83% for Step AG-CoT, 2.21% and 5.26% for GRPO-v2, and 3.13% and 6.14% for RFT-v1. Between 49% and 58% of the outputs that prepare the target state are thus rejected by label matching. With 64 candidates per target, coverage over the same 249,216 Basic AG-CoT candidates is 4.42% under label matching and 10.07% under state equivalence (Table 4, p9).
W4 + Q5. Evaluation scope and structured error-correction tasks
Reviewer concern. All empirical evaluations are restricted solely to Clifford random-circuit benchmarks derived from QCircuitBench, lacking evaluation on broader non-Clifford or realistic physical quantum error correction circuits.
In Section 6, evaluation is restricted to random Clifford targets. How does the AG-CoT approach perform when applied to structured quantum error correction benchmarks such as StabilizerBench?
Response.
The comparison needs an exact check on every output and a step-by-step derivation of a correct circuit for every target. Clifford state preparation provides both (§1, p1), which is why the experiments use random Clifford targets (§5, p6).
A code state fixed by n generators, such as a logical basis state, is a stabilizer state, so the same teacher construction and the same verifier apply to it. StabilizerBench specifies a code by fewer than n generators, so its targets are codespaces. Fixing a logical state gives a target that our pipeline can handle.
For non-stabilizer targets, §3 (p3) states what the extension requires, a representation for non-stabilizer states and a fidelity threshold fixed before evaluation.
I would like to thank the authors for the responses. In light of the rebuttal and other reviews, I keep my score.
Response to Reviewer bfjw (1/3)
W1 + Q1. Motivation for algorithmic trace supervision
Response.
We thank the reviewer for the detailed questions, and for recognizing the gain over circuit-only training across model scales and the compact target representation.
Standard fine-tuning does learn this representation, to a limited degree. The paper measures three approaches on this task.
A general-purpose chain of thought has no procedure that generates it for each target and no exact check that can replay it. Each AG trace is produced by the same deterministic procedure as the teacher circuit, which the exact check accepts (§4, p5), and its steps are exact tableau updates that can be replayed and checked (§6.3, p10).
The three numbered points are now stated in §2 (p3).