- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SynQBench, a synthetic benchmark for neural quantum error correction decoders based on hardware data. It initializes circuit-level Pauli noise from device profiles, calibrates selected parameters against hardware detector-event and correlation statistics, and generates labeled syndrome data with Stim. The evaluation focuses on Willow's distance-5, ten-round, Z-basis memory configuration. The authors compare simulated statistics with held-out hardware data and study synthetic training and evaluation with Transformer, GNN, CNN, and MLP decoders, alongside a fixed MWPM reference. They report better hardware decoding than with default synthetic noise models and substantially smaller gaps between synthetic and hardware logical error rates.
Critical strengths
-
Section 4 evaluates detector rates and correlations using 10,000 hardware shots excluded from calibration and update acceptance (Appendix C). This provides independent evidence that the fitted simulator captures hardware statistics beyond the samples used for fitting. Connecting these statistics to decoder performance addresses a practical question for researchers who rely on synthetic QEC data.
-
Training protocols are controlled within each architecture, and checkpoints are selected solely by synthetic validation loss. All 16 selected neural checkpoints are evaluated on all four synthetic distributions, providing useful evidence for a shared benchmark. In the comparison between the base and calibrated versions, the mean absolute gap between synthetic and hardware LER falls from approximately 1.186 to 0.288 percentage points, a reduction by a factor of about 4.1 (Figure 5; Tables 10-13). This is a substantial improvement under the reported protocol, although independent downstream validation is still needed.
Critical weaknesses
-
The downstream hardware evaluation reuses data that helped construct the simulator, so it is not independent of the full pipeline. Appendices C and D.3-D.4 state that Section 5 uses all 50,000 hardware shots, including 30,000 calibration shots and 10,000 acceptance shots. Thus, 80% of the evaluation records influenced construction of the generator that supplies the synthetic training and evaluation data. Selecting checkpoints solely by synthetic validation loss does not remove this dependence. The results leave open whether the training gains and smaller evaluation gaps persist on hardware observations excluded from simulator construction. Re-evaluating the frozen checkpoints on the existing held-out subset of 10,000 shots would directly address this concern without retraining.
-
The comparisons do not establish how much SynQBench gains beyond matching the overall noise level. Uniform and SI1000 both use a fixed global parameter of , and Figure 2 shows that both substantially underestimate hardware detector-event rates. The comparison therefore does not separate the benefits of location-specific modeling and statistics-guided calibration from those of a better overall noise setting. SynQBench-base is a useful ablation, but it does not replace a simple baseline calibrated to the overall noise level. This limits the assessment of the benchmark's incremental value. Fitting SI1000's global parameter using only the permitted calibration data would provide a direct control.
Questions for authors
-
With the generator and selected checkpoints held fixed, what are the hardware LERs and Figure 5 evaluation gaps on the 10,000-shot subset excluded from simulator construction? Do the main conclusions hold with uncertainty estimates that preserve the paired structure of predictions on the same hardware shots?
-
How much advantage remains after fitting SI1000's global noise parameter on the permitted calibration/acceptance splits? An evaluation with a fixed decoder and one representative neural training comparison would help separate noise-scale matching from the additional value of the proposed construction.
-
Which fidelity and depolarization conventions are used in Appendix A.2, Eqs. (8) and (9)? In particular, why are the two coherence fidelities multiplied, and how are the resulting two-qubit channel and Stim probabilities derived consistently? Please distinguish average gate fidelity from process/entanglement fidelity.
-
How many independent training runs were used for each condition? Table 9 gives Wilson intervals, whereas Tables 10-13 report point estimates. Please distinguish sampling uncertainty for each checkpoint from variability between training runs, and quantify uncertainty for the main paired comparisons and aggregate evaluation gaps, especially the smaller base-to-calibrated training differences.
6 comments
Q3. Fidelity conventions in Appendix A.2. This detailed question led us to re-examine our fidelity conventions and refine the implementation; the revised Appendix A.2 now states them explicitly:
- Fidelity type. All fidelities in Eq. (12), including and , are average gate fidelities.
- Two-qubit coherence fidelity. Each qubit decoheres independently during the gate, so the coherence channel is . Entanglement (process) fidelity is multiplicative under tensor products, whereas average fidelity is not. "The product of the two coherence fidelities" therefore refers to the entanglement fidelities, . This product is converted back to an average fidelity via with .
- Depolarizing channel. In Eq. (12), is the mixing parameter of . It is chosen so that composing the / channel with this depolarizing channel reproduces , so coherence loss is not counted twice. When , is set to 0.
- Stim probabilities. Stim's
DEPOLARIZE1/DEPOLARIZE2(p)take the total non-identity Pauli probability, so we pass , i.e., and . The / channel is simulated as its Pauli twirl, which has the same average fidelity.
Thanks for this constructive question. Auditing these conventions also led us to refine two of them in the model: the two-qubit coherence fidelity and the probabilities passed to Stim. All SynQBench results in the response and revision use the refined model.
Q4 (General Response, Points 2 and 3). Training runs and uncertainty. We appreciate the reviewer's attention to training variability and address it below.
Within the discussion period, we retrained the full suite of 24 decoders (six training distributions × four architectures) under a single protocol, with one run per condition; the intervals in Q1 quantify shot-sampling uncertainty for each run. Because repeating the full suite multiple times was not feasible in this period, we assessed run-to-run variability through repeated Transformer training and consistency checks across architectures and checkpoints. The results show:
- Gaps exceed the run-to-run spread. Three independent Transformer runs on an earlier version of the SynQBench data reached 2.53%, 2.64%, and 2.83% on the same test shots. Their 0.30 pp spread is smaller than every gap to the calibrated baselines (0.39–0.75 pp).
- Evidence beyond a single LER. The training value does not rest on one aggregate LER per run: the shot-level analyses in General Response, Point 3 (paired outcomes, difficulty-stratified errors, per-stabilizer comparisons, and prediction confidence) consistently favor SynQBench across all four architectures.
The paired comparisons on the held-out shots below separate the gains over the calibrated baselines from the gain of calibration within SynQBench, and provide further evidence of SynQBench's training value beyond a single LER (McNemar with Holm over 20 comparisons; log-loss with paired 95% intervals):
| SynQBench vs | Willow LER lower | LER difference significant | Log-loss lower, interval excludes 0 |
|---|---|---|---|
| Uniform-cal, SI1000-cal | 8 of 8 (0.39–0.75 pp) | 8 of 8 (adjusted ) | 8 of 8 (12.9–19.9%) |
| SynQBench-base | 4 of 4 (0.07–0.34 pp) | 1 of 4 (GNN, adjusted ) | 4 of 4 (8.4–10.5%) |
- Over the calibrated baselines, the gains are significant in every architecture and hold even against each baseline's best held-out epoch (General Response, Point 2(b)).
- Over SynQBench-base, calibration lowers LER in all four architectures, significantly only for GNN. Its benefit is clearer in log-loss (all four) and in evaluation: mean gap 1.40 → 0.23 pp, Spearman 0.86 → 0.97 (General Response, Point 2).
Thank you for the response. The held-out evaluation and the fitted SI1000 baseline address my two main concerns about the evaluation protocol.
On Q4, the response still does not report how many independent training runs were performed per condition, so I can’t tell how much the results vary across seeds. I applied McNemar’s test to the paired counts without adjusting for multiple comparisons. The difference between SynQBench and SynQBench-base was statistically significant only for GNN (100/66). The differences for Transformer (90/72), CNN (86/79), and MLP (126/105) were not statistically significant. That leaves me uncertain about the downstream benefit of calibration, which is the main addition beyond PAEMS.
Your response mentions a refined noise model in connection with Q3. The question of which fidelity conventions you used and what changed in the model remains unanswered.
All the validation uses a single Willow configuration (d = 5, 10 rounds, Z basis). That limits the evidence for presenting SynQBench as a standardized benchmark.
I’m keeping my original score.
We thank the reviewer for the precise and actionable questions, which directly refined the new experimental protocol. The shared results are in the General Response; below we add only what is specific to each question.
Q1 (General Response, Point 1). Held-out evaluation with paired uncertainty.
All 24 decoders were retrained on the refined noise model (Q3), selected by synthetic validation loss, and evaluated on the 10,000 held-out shots excluded from calibration and acceptance; every conclusion holds. We report results from this updated suite in place of the original submission's checkpoints, which were trained on the legacy model.
The uncertainty analysis preserves the paired structure of the shots at three levels:
Q2 (General Response, Points 2 and 3). SI1000 with a fitted global parameter. We fitted p only on the permitted 30,000 calibration shots, by minimizing the detector-event-rate MAE (p=0.0025 for SI1000). Both requested controls follow:
Matching the noise scale explains part of the improvement over SI1000-base; the location-specific construction explains the rest. Fitting p to enhance the baseline leaves the rate pattern (r=0.71) and the decoder ranking (Spearman 0.49) essentially unchanged, and SynQBench reaches 0.92 and 0.97.