- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
In quantum circuit generation, directly evaluating candidates against references could reject semantically correct solutions. To address this gap, the authors propose QuARC, a benchmark of 86 tasks with acceptance regions. The acceptance region includes four equivalent transformations: global phase, gate decomposition, qubit relabeling, ancillary extension. The paper includes theoretical analysis of acceptance region for Haar random unitaries. Experiments are conducted over 15 models, with case analysis on where acceptance region takes effect.
Critical strengths
The overall idea of including equivalent transformations in quantum circuit generation to prevent false rejection of semantically correct solutions is valuable.
Critical weaknesses
- As a benchmark, the paper lacks justification for task selection, their representativeness for quantum circuit generation, what verifications they need specifically, and how the acceptance regions directly connect with these tasks. I suggest at least add a summary of Appendix A in the main text. The theoretical analysis of Haar random unitaries is almost irrelevant to the practical evaluation, as LLM-generated circuits are almost never Haar random in practice. The benchmark provides limited discrimination among leading models and therefore has a problem of saturation. The limitation section includes many of my concerns, e.g. flawed prompts, which I would not repeat here.
- The overall writing quality could be greatly improved. The paper includes too much technical details and qualifications, but overlooks important aspects such as benchmark construction. The language is hard to understand too.
Questions for authors
Could you give a detailed explanation for how evaluators from previous benchmarks misjudge correct solutions, how often does this happen, and how QuARC solves this problem? For example, QCircuitBench gives detailed requirements in the prompt to ensure the uniqueness of output, and evaluators accept semantically correct solutions without requiring them to be identical to the reference implementation. What's the advantage of QuARC over it?