acceptodds
Under review as a conference paper at ICLR 2027

SynQBench: Hardware-Grounded Synthetic Benchmarks for Neural Quantum Error Correction Decoder

Abstract

Neural decoders for quantum error correction (QEC) rely heavily on synthetic data for both training and evaluation. However, the field currently lacks standardized, hardware-grounded benchmarks. Consequently, existing studies commonly depend on custom datasets with simplified noise assumptions, compromising the data quality and limiting the hardware relevance of resulting conclusions. In this work, we introduce SynQBench, a hardware-grounded synthetic benchmark for neural QEC decoders. SynQBench integrates device-dependent physical profiles and empirical calibrations into a circuit-aware noise model, enabling the scalable generation of faithful, labeled syndrome data. We validate the statistical fidelity of SynQBench against Google's Willow processor measurements, achieving a reduction in the mean absolute error of detector-event rates relative to the widely used SI1000 noise model, alongside substantially closer agreement with hardware spatial and temporal correlations. Furthermore, we demonstrate dual downstream value of SynQBench across the decoder development lifecycle: training on SynQBench reduces hardware logical error rates by to across four decoder architectures, while evaluating on SynQBench narrows the sim-to-real gap in logical error rate by across 16 trained decoders. Together, these results position SynQBench as a standardized, hardware-grounded foundation for reproducible neural QEC decoder development and assessment.

Primary area:
datasets and benchmarks
Keywords:
Quantum Error Correction, Neural Decoder, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DgXcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SynQBench, a synthetic benchmark for neural quantum error correction decoders based on hardware data. It initializes circuit-level Pauli noise from device profiles, calibrates selected parameters against hardware detector-event and correlation statistics, and generates labeled syndrome data with Stim. The evaluation focuses on Willow's distance-5, ten-round, Z-basis memory configuration. The authors compare simulated statistics with held-out hardware data and study synthetic training and evaluation with Transformer, GNN, CNN, and MLP decoders, alongside a fixed MWPM reference. They report better hardware decoding than with default synthetic noise models and substantially smaller gaps between synthetic and hardware logical error rates.

Critical strengths
  1. Section 4 evaluates detector rates and correlations using 10,000 hardware shots excluded from calibration and update acceptance (Appendix C). This provides independent evidence that the fitted simulator captures hardware statistics beyond the samples used for fitting. Connecting these statistics to decoder performance addresses a practical question for researchers who rely on synthetic QEC data.

  2. Training protocols are controlled within each architecture, and checkpoints are selected solely by synthetic validation loss. All 16 selected neural checkpoints are evaluated on all four synthetic distributions, providing useful evidence for a shared benchmark. In the comparison between the base and calibrated versions, the mean absolute gap between synthetic and hardware LER falls from approximately 1.186 to 0.288 percentage points, a reduction by a factor of about 4.1 (Figure 5; Tables 10-13). This is a substantial improvement under the reported protocol, although independent downstream validation is still needed.

Critical weaknesses
  1. The downstream hardware evaluation reuses data that helped construct the simulator, so it is not independent of the full pipeline. Appendices C and D.3-D.4 state that Section 5 uses all 50,000 hardware shots, including 30,000 calibration shots and 10,000 acceptance shots. Thus, 80% of the evaluation records influenced construction of the generator that supplies the synthetic training and evaluation data. Selecting checkpoints solely by synthetic validation loss does not remove this dependence. The results leave open whether the training gains and smaller evaluation gaps persist on hardware observations excluded from simulator construction. Re-evaluating the frozen checkpoints on the existing held-out subset of 10,000 shots would directly address this concern without retraining.

  2. The comparisons do not establish how much SynQBench gains beyond matching the overall noise level. Uniform and SI1000 both use a fixed global parameter of p=0.001p = 0.001, and Figure 2 shows that both substantially underestimate hardware detector-event rates. The comparison therefore does not separate the benefits of location-specific modeling and statistics-guided calibration from those of a better overall noise setting. SynQBench-base is a useful ablation, but it does not replace a simple baseline calibrated to the overall noise level. This limits the assessment of the benchmark's incremental value. Fitting SI1000's global parameter using only the permitted calibration data would provide a direct control.

Questions for authors
  1. With the generator and selected checkpoints held fixed, what are the hardware LERs and Figure 5 evaluation gaps on the 10,000-shot subset excluded from simulator construction? Do the main conclusions hold with uncertainty estimates that preserve the paired structure of predictions on the same hardware shots?

  2. How much advantage remains after fitting SI1000's global noise parameter on the permitted calibration/acceptance splits? An evaluation with a fixed decoder and one representative neural training comparison would help separate noise-scale matching from the additional value of the proposed construction.

  3. Which fidelity and depolarization conventions are used in Appendix A.2, Eqs. (8) and (9)? In particular, why are the two coherence fidelities multiplied, and how are the resulting two-qubit channel and Stim probabilities derived consistently? Please distinguish average gate fidelity from process/entanglement fidelity.

  4. How many independent training runs were used for each condition? Table 9 gives Wilson intervals, whereas Tables 10-13 report point estimates. Please distinguish sampling uncertainty for each checkpoint from variability between training runs, and quantify uncertainty for the main paired comparisons and aggregate evaluation gaps, especially the smaller base-to-calibrated training differences.

6 comments

We thank the reviewer for the precise and actionable questions, which directly refined the new experimental protocol. The shared results are in the General Response; below we add only what is specific to each question.

Q1 (General Response, Point 1). Held-out evaluation with paired uncertainty.
All 24 decoders were retrained on the refined noise model (Q3), selected by synthetic validation loss, and evaluated on the 10,000 held-out shots excluded from calibration and acceptance; every conclusion holds. We report results from this updated suite in place of the original submission's checkpoints, which were trained on the legacy model.

The uncertainty analysis preserves the paired structure of the shots at three levels:

  • Per checkpoint: 95% Wilson intervals (±0.3–0.5 pp), now included in the per-decoder tables.
  • Paired LER comparisons: exact two-sided McNemar tests on the discordant shots, Holm-corrected over the 20 within-architecture comparisons.
  • Aggregate gaps and rank correlations: paired-shot bootstrap intervals (2,000 replicates) that resample the test shots jointly across all decoders, holding the 1M-shot synthetic LERs fixed.

Q2 (General Response, Points 2 and 3). SI1000 with a fitted global parameter. We fitted pp only on the permitted 30,000 calibration shots, by minimizing the detector-event-rate MAE (p=0.0025p=0.0025 for SI1000). Both requested controls follow:

  • Fixed decoder: the MWPM sim-to-real gap is 2.26 pp on SI1000-cal and 0.04 pp on SynQBench.
  • Neural comparison: training on SynQBench lowers Willow LER by 0.56–0.71 pp in all four architectures (adjusted p≤4.8×10−4p\le 4.8\times10^{-4}).

Matching the noise scale explains part of the improvement over SI1000-base; the location-specific construction explains the rest. Fitting pp to enhance the baseline leaves the rate pattern (r=0.71r=0.71) and the decoder ranking (Spearman 0.49) essentially unchanged, and SynQBench reaches 0.92 and 0.97.

Q3. Fidelity conventions in Appendix A.2. This detailed question led us to re-examine our fidelity conventions and refine the implementation; the revised Appendix A.2 now states them explicitly:

  • Fidelity type. All fidelities in Eq. (12), including FhwF^{\mathrm{hw}} and FcohF^{\mathrm{coh}}, are average gate fidelities.
  • Two-qubit coherence fidelity. Each qubit decoheres independently during the gate, so the coherence channel is Pi⊗Pj\mathcal{P}_i\otimes\mathcal{P}_j. Entanglement (process) fidelity is multiplicative under tensor products, whereas average fidelity is not. "The product of the two coherence fidelities" therefore refers to the entanglement fidelities, Fe,ij=Fe,iFe,jF_{e,ij}=F_{e,i}F_{e,j}. This product is converted back to an average fidelity via Favg=(nFe+1)/(n+1)F_{\mathrm{avg}}=(nF_e+1)/(n+1) with n=4n=4.
  • Depolarizing channel. In Eq. (12), rr is the mixing parameter of ρ↦(1−r)ρ+rI/n\rho\mapsto(1-r)\rho+r I/n. It is chosen so that composing the T1T_1/T2T_2 channel with this depolarizing channel reproduces FhwF^{\mathrm{hw}}, so coherence loss is not counted twice. When Fcoh<FhwF^{\mathrm{coh}}<F^{\mathrm{hw}}, rr is set to 0.
  • Stim probabilities. Stim's DEPOLARIZE1/DEPOLARIZE2(p) take the total non-identity Pauli probability, so we pass p=r(1−1/n2)p=r(1-1/n^2), i.e., 3r/43r/4 and 15r/1615r/16. The T1T_1/T2T_2 channel is simulated as its Pauli twirl, which has the same average fidelity.

Thanks for this constructive question. Auditing these conventions also led us to refine two of them in the model: the two-qubit coherence fidelity and the probabilities passed to Stim. All SynQBench results in the response and revision use the refined model.

Q4 (General Response, Points 2 and 3). Training runs and uncertainty. We appreciate the reviewer's attention to training variability and address it below.

Within the discussion period, we retrained the full suite of 24 decoders (six training distributions × four architectures) under a single protocol, with one run per condition; the intervals in Q1 quantify shot-sampling uncertainty for each run. Because repeating the full suite multiple times was not feasible in this period, we assessed run-to-run variability through repeated Transformer training and consistency checks across architectures and checkpoints. The results show:

  • Gaps exceed the run-to-run spread. Three independent Transformer runs on an earlier version of the SynQBench data reached 2.53%, 2.64%, and 2.83% on the same test shots. Their 0.30 pp spread is smaller than every gap to the calibrated baselines (0.39–0.75 pp).
  • Evidence beyond a single LER. The training value does not rest on one aggregate LER per run: the shot-level analyses in General Response, Point 3 (paired outcomes, difficulty-stratified errors, per-stabilizer comparisons, and prediction confidence) consistently favor SynQBench across all four architectures.

The paired comparisons on the held-out shots below separate the gains over the calibrated baselines from the gain of calibration within SynQBench, and provide further evidence of SynQBench's training value beyond a single LER (McNemar with Holm over 20 comparisons; log-loss with paired 95% intervals):

SynQBench vsWillow LER lowerLER difference significantLog-loss lower, interval excludes 0
Uniform-cal, SI1000-cal8 of 8 (0.39–0.75 pp)8 of 8 (adjusted p≤0.025p\le 0.025)8 of 8 (12.9–19.9%)
SynQBench-base4 of 4 (0.07–0.34 pp)1 of 4 (GNN, adjusted p=0.041p=0.041)4 of 4 (8.4–10.5%)
  • Over the calibrated baselines, the gains are significant in every architecture and hold even against each baseline's best held-out epoch (General Response, Point 2(b)).
  • Over SynQBench-base, calibration lowers LER in all four architectures, significantly only for GNN. Its benefit is clearer in log-loss (all four) and in evaluation: mean gap 1.40 → 0.23 pp, Spearman 0.86 → 0.97 (General Response, Point 2).

Thank you for the response. The held-out evaluation and the fitted SI1000 baseline address my two main concerns about the evaluation protocol.

On Q4, the response still does not report how many independent training runs were performed per condition, so I can’t tell how much the results vary across seeds. I applied McNemar’s test to the paired counts without adjusting for multiple comparisons. The difference between SynQBench and SynQBench-base was statistically significant only for GNN (100/66). The differences for Transformer (90/72), CNN (86/79), and MLP (126/105) were not statistically significant. That leaves me uncertain about the downstream benefit of calibration, which is the main addition beyond PAEMS.

Your response mentions a refined noise model in connection with Q3. The question of which fidelity conventions you used and what changed in the model remains unanswered.

All the validation uses a single Willow configuration (d = 5, 10 rounds, Z basis). That limits the evidence for presenting SynQBench as a standardized benchmark.

I’m keeping my original score.

Reviewer_zTqSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SynQBench builds on the PAEMS framework for hardware-aware, circuit-level noise modeling. It uses device parameters and measured syndrome statistics to calibrate a noise model, then generates labeled synthetic data with Stim for supervised training of neural quantum error correction decoders. The paper introduces a structured calibration procedure and evaluates the resulting data through syndrome statistics and decoder performance on Willow measurements.

Critical strengths
  1. The paper implements a complete workflow from hardware measurements to synthetic data and neural decoder training. Connecting these components provides a practical setting for studying how the choice of synthetic training data affects decoding on real hardware. The value here is the integration and its evaluation, rather than the individual components.

  2. The structured calibration procedure is concrete and clearly specified. Grouping parameters by their physical roles and checking proposed updates on a separate acceptance split gives the calibration process a clear organization. This is a potentially useful contribution, although its advantage over existing calibration procedures remains to be established.

Critical weaknesses
  1. The methodological contribution beyond PAEMS is not sufficiently established. As presented, the work largely combines PAEMS-style noise modeling with standard supervised training of neural decoders, together with a structured calibration procedure. PAEMS already calibrates noise models using real experimental data, so hardware calibration itself does not distinguish this work. The paper needs to identify its new technical elements more clearly and demonstrate their value through direct comparisons and targeted ablations.

  2. The evaluation protocol and the interpretation of some results raise concerns. The final downstream hardware evaluation includes records used to calibrate the simulator, weakening its independence as a test of performance on unseen hardware data. In addition, SI1000 uses a fixed global noise parameter, without a comparison in which that parameter is fitted using the same hardware calibration data. This leaves open how much of the improvement requires the proposed modeling approach rather than a better choice of overall noise strength. Finally, closer agreement between simulated and hardware decoder error rates is not, by itself, sufficient to establish that SynQBench captures Willow’s underlying noise more accurately than SI1000.

Questions for authors
  1. What specific changes does SynQBench introduce beyond PAEMS in noise modeling and calibration? Could the authors provide a direct comparison and targeted ablations to establish the contribution of the structured calibration procedure?

  2. The downstream hardware evaluation includes records used for simulator calibration. Could the authors repeat this evaluation using hardware records excluded from both calibration and update acceptance? Do the training advantage and the closer agreement between simulated and hardware error rates persist?

  3. If SI1000’s global noise parameter is fitted using the same hardware calibration data, how much of SynQBench’s advantage remains in statistical agreement and hardware decoding performance?

  4. Closer agreement between simulated and hardware error rates for the tested decoders does not by itself establish that SynQBench models Willow’s real noise more accurately than SI1000. Can the authors provide additional experimental or theoretical evidence to support this conclusion?

3 comments

We sincerely thank the reviewer for their thorough evaluation and perceptive comments. Every point raised touches on a critical aspect of our work, and addressing them has markedly strengthened the quality of the manuscript. The shared results are in the General Response; below we add only what is specific to each question.

Q1 (General Response, Point 4). Contribution beyond PAEMS, and targeted ablations. In short, SynQBench extends PAEMS (He et al., 2026), a noise model fitted to repetition-code experiments, into a benchmark for neural decoders on surface-code data.

As targeted ablations, we pair each model family with default or calibrated parameters. All results are on held-out shots:

AblationStatisticsDownstream
Calibration of a homogeneous model (Uniform/SI1000-base → -cal)DER MAE 4.1–4.2 → 1.1–1.2 pp, but DER rr and matrix rr unchangedGap 3.65 → 0.75 and 3.72 → 1.99 pp; ranking unchanged
Calibration of the hardware-aware model (SynQBench-base → SynQBench)DER rr 0.82 → 0.92; spatial / temporal / spacetime residuals 3.2× / 7.1× / 1.1× lowerGap 1.40 → 0.23 pp; rank correlation 0.86 → 0.97; LER lower in all architectures

We did not rerun PAEMS's black-box evolutionary fitting on Willow, because it was designed for, and conducted on, repetition-code experiments on other processors (IBM and Chinese cloud devices); its fitting targets do not cover the two stabilizer types and spacetime correlations of the surface code. Its parameterization also differs: PAEMS fits every qubit parameter, whereas SynQBench updates three physically grouped parameter blocks. The ablations above therefore evaluate each component of SynQBench, including its calibration flow, within one model and one fixed protocol.

Q2 (General Response, Point 1). Evaluation on records excluded from calibration and acceptance. Section 4 already used such held-out shots. Section 5 now uses them as well, and both the training advantage and the closer LER agreement persist.

Q3 (General Response, Points 2 and 3). Advantage remaining after fitting SI1000's global parameter. The advantage remains substantial in both respects.

  • Statistical agreement. SI1000-cal matches Willow's average detector-event rate, but its rate pattern stays at r=0.71r=0.71, against 0.92 for SynQBench. Its temporal correlation residual is 12.5× SynQBench's.
  • Downstream value. On training, decoders trained on SynQBench reach 0.56–0.71 pp (16.8–18.5%) lower Willow LER than those trained on SI1000-cal in all four architectures. On evaluation, SI1000-cal still misestimates hardware performance: its mean sim-to-real gap is 1.99 pp (SynQBench: 0.23 pp, 8.8× smaller), and its decoder ranking agrees with Willow's at Spearman 0.49 (SynQBench: 0.97), misordering 52 of 152 decoder pairs (8 for SynQBench, 6.5× fewer).

Q4 (General Response, Point 2). Evidence beyond agreement in LER. We agree that matching LER alone does not establish that a simulator models the hardware's noise. Section 4 is specifically designed to address this by statistically comparing synthetic and hardware syndromes, independent of any decoder, across three levels of granularity on the held-out shots. Statistics of the calibration-enhanced baselines are also included in the revision.

Together with decoder rank agreement, we provide four lines of evidence:

  1. Detector-event rates: SynQBench reproduces the pattern, not only the average. Calibrating pp for Uniform and SI1000 matches Willow's average rate but rescales all rates together, leaving their pattern unchanged (Pearson rr 0.71–0.72). SynQBench reaches r=0.92r=0.92.
  2. Structured correlations: 3–5× lower residuals. SynQBench's spatial and temporal correlation residuals are 3.1× and 5.1× lower than those of the best calibrated baseline. For SI1000, fitting pp more than doubles the temporal residual.
  3. Full correlation matrix: closer agreement. SynQBench agrees with Willow at r=0.88r=0.88, against 0.73–0.81 for the calibrated baselines, values that calibration leaves unchanged.
  4. Decoder ranking: SynQBench orders decoders as the hardware does. Across 18 decoders, its ranking agrees with Willow's at Spearman 0.97, against 0.49–0.69 for the calibrated baselines. Calibration does not improve their rankings.

Thank you again for your review. As the discussion period closes today (Nov 18, AoE), we would like to check whether our response addresses your concerns. We would be happy to clarify anything further, and we would be grateful if you could take these results into account in your final assessment.

In brief, we added calibrated Uniform and SI1000 baselines and evaluated all 24 retrained decoders on held-out Willow shots: SynQBench still lowers hardware LER by 0.39–0.75 pp in all four architectures, and its sim-to-real gap is 3.3× and 8.8× smaller than those of the calibrated baselines. Our reply to you also includes targeted calibration ablations (Q1) and decoder-independent evidence beyond LER agreement (Q4). You can see more details there and in shared general responses.

Thank you again for your thorough review. With less than 12 hours left in the discussion period, we would like to check whether our response addresses your questions. We would be happy to clarify anything further and would be grateful if you could take these responses into account in your final assessment.

Reviewer_JMdfOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SynQBench, a hardware-grounded synthetic benchmark for neural quantum error correction decoders. The framework combines device-specific noise modeling with syndrome statistics-based calibration to generate realistic synthetic training and evaluation data using Stim. Experiments on Google's Willow processor demonstrate improved agreement with hardware detector statistics. Training on SynQBench improves hardware decoding performance across four neural decoder architectures and substantially reduces the sim-to-real LER gap.

Critical strengths
  • The proposed framework provides a useful approach to generating hardware-grounded synthetic QEC data by combining device-specific noise information with syndrome-based calibration.

  • The method achieves strong empirical results, improving hardware decoding performance across multiple neural architectures and providing more accurate estimates of real hardware LER.

Critical weaknesses
  • The methodological novelty is somewhat limited, as the hardware-aware noise modeling largely follows the existing PAEMS framework. Furthermore, the main comparisons use uncalibrated SI1000 and Uniform noise models, making it difficult to isolate the additional benefit of the proposed approach.
Questions for authors

As I am not sufficiently familiar with quantum hardware and hardware-specific noise modeling, I would defer to the assessments of other reviewers with more expertise in this area.

3 comments

We thank the reviewer for the positive assessment and are glad that the reviewer found our paper informative. Both concerns raised are well taken and align with feedback from other reviewers; we have addressed them in detail in our General Response.

Concern 1 (General Response, Point 4). Contribution relative to PAEMS. PAEMS (He et al., 2026) is a qubit-resolved noise model fitted to repetition-code experiments. SynQBench builds its device-informed initialization on this channel construction and goes beyond it in four respects that PAEMS does not cover:

  • from a noise model to a benchmark for neural decoders, validated for both training and evaluation;
  • from repetition codes to the surface code, validated on real Willow hardware data;
  • a statistics-guided calibration that generalizes to held-out hardware data, rather than a black-box search;
  • hardware profiles for platforms beyond those PAEMS studied.

Concern 2 (General Response, Points 2 and 3). Calibration-enhanced baselines. During the discussion period, we fitted the SI1000 and Uniform baselines on the calibration shots and retrained all 24 decoders. SynQBench's advantage remains clear in both statistical agreement and downstream value.

Thank you again for your review. As the discussion period closes today (Nov 18, AoE), we would like to check whether our response addresses your concerns. We would be happy to clarify anything further, and we would be grateful if you could take these results into account in your final assessment.

In brief, we added calibrated Uniform and SI1000 baselines and evaluated all 24 retrained decoders on held-out Willow shots: SynQBench still lowers hardware LER by 0.39–0.75 pp in all four architectures, and its sim-to-real gap is 3.3× and 8.8× smaller than those of the calibrated baselines. General Response, Point 4 also summarizes SynQBench's contributions beyond PAEMS.

Thank you again for your positive assessment. With less than 12 hours left in the discussion period, we would like to check whether our response addresses your two concerns. We would be grateful if you could take our responses into account in your final assessment!

Comments on the paper

5 comments
AuthorsOpenReview ↗

General Response Part One

Thank you for your careful evaluation and insightful feedback. Your comments have greatly helped us improve the experimental soundness of SynQBench and strengthen the empirical evidence supporting its value.

[Overview of General Response]

The reviews share two concerns about soundness: the independence of the downstream hardware evaluation, and the strength of the SI1000 and Uniform baselines. They also share a question about the relation to PAEMS. In response, we:

  • added calibrated versions of both baselines and retrained all 24 decoders;
  • evaluated all decoders on held-out hardware shots;
  • added detailed shot-level analysis of the decoders along with deeper analytical insights.

Where each question is answered.

General ResponseQuestions
Point 1: held-out evaluationzTqS Q2; DgXc Q1
Point 2: calibrated baselinesJMdf Concern 2; zTqS Q3, Q4; DgXc Q2, Q4
Point 3: shot-level analysesJMdf Concern 2; zTqS Q3; DgXc Q2, Q4
Point 4: relation to PAEMSJMdf Concern 1; zTqS Q1

The individual replies add only reviewer-specific material: the ablation table (zTqS Q1), the fidelity conventions (DgXc Q3), and training-run uncertainty (DgXc Q4).

Notation.

  • Uniform-base and SI1000-base are the submission's baselines at p=0.001p=0.001. This is the conventional operating point for superconducting-inspired circuit noise, where the value matches the SI1000 prior distributed with Google Quantum AI's public surface-code dataset.
  • Uniform-cal and SI1000-cal use the same models with the global pp fitted on the same 30,000 calibration shots used for our model (yielding p=0.00257p = 0.00257 and p=0.0025p = 0.0025, respectively). The fit minimizes the mean absolute error of the detector-event rates, the statistic that a global scale controls.
  • SynQBench-base and SynQBench are our hardware-aware model before and after statistics-guided calibration.

1. Downstream evaluation soundness: all hardware results are now computed on held-out shots, and every headline claim of the submission holds

We split the 50,000 Willow shots into 30,000 calibration, 10,000 acceptance, and 10,000 test shots. The test shots were held out and never used for calibration, acceptance, checkpoint selection, or early stopping. Checkpoints are selected only by synthetic validation loss. Section 4 already used the held-out shots; Section 5 now does as well.

We thank Reviewer DgXc for the detailed and constructive comments on our noise modeling, based on which we further refined our model (detailed in the individual response).

We then retrained all 24 decoders, both to add the calibrated baselines and to record performance on the held-out shots at every epoch for the trajectory analysis in Point 2(b). These records informed no modeling, training, or selection decision; each decoder is selected only by synthetic validation loss. All results below come from these retrained decoders to ensure a fair and comprehensive comparison.

The table below lists the held-out Willow LER of all 24 decoders: SynQBench still yields the lowest LER for every architecture, and all four SynQBench-trained decoders outperform the fixed MWPM reference (4.23%).

Willow LER on the held-out shots (%).

Training distributionTransformerGNNCNNMLP
Uniform-base3.824.614.187.41
SI1000-base3.904.914.366.63
Uniform-cal3.263.203.104.27
SI1000-cal3.253.263.324.23
SynQBench-base2.833.042.783.73
SynQBench2.652.702.713.52

Re-evaluating the submission's headline claims with the retrained decoders on the held-out shots confirms all of them, with margins equal to or larger than those reported (summarized in the table below). All 24 decoders are scored on the same 10,000 held-out shots under the same protocol, so every comparison fairly uses identical test data and settings.

Claims of the submission under the held-out protocol.

ClaimSubmission (all 50,000 shots)Revision (10,000 held-out shots)
Detector-event-rate MAE, reduction vs SI1000-base6.9×8.2×
Willow LER reduction from training on SynQBench vs Uniform-base / SI1000-base28.1–47.1% / 30.4–42.0%30.6–52.5% / 32.1–46.9%
Mean sim-to-real gap, reduction vs Uniform-base / SI1000-base (16 decoders)13.7× / 14.0×17.5× / 17.8×
AuthorsOpenReview ↗

General Response Part Two

2. Calibration-enhanced baselines: noise scale matched, structure not; SynQBench's advantage persists from statistics to downstream

Calibrating the Uniform and SI1000 baselines raises their average detector-event rate to Willow's level. However, it leaves the spatial noise pattern of rates and correlations as mismatched as before. The same pattern appears downstream: calibration lowers the hardware error of the trained decoders, but it does not reproduce how the hardware ranks them.

(a) Statistical agreement

Table columns:

  • DER: detector-event rates (MAE in pp, and Pearson$^*$ rr of their pattern);
  • MAR: mean absolute residual of the spatial, temporal, and spacetime correlations (×10⁻³);

* Importantly, Pearson rr measures how well a model reproduces Willow's spatial pattern of detector statistics, independent of their overall scale; values closer to 1 indicate a closer match to the hardware.

Seed-to-seed 95% intervals of the MAR are at most 0.13×10⁻³.

ModelDER MAEDER rrSpatial MARTemporal MARSpacetime MAR
Uniform-base4.220.7211.007.561.93
Uniform-cal1.100.724.983.951.58
SI1000-base4.110.7111.554.292.51
SI1000-cal1.150.716.419.781.54
SynQBench-base0.810.825.085.551.22
SynQBench0.500.921.600.781.07
  • Rate scale vs. rate pattern. Within the Uniform and SI1000 baselines, fitting pp lowers the DER MAE by 3.6–3.8×, but the Pearson rr of the rate pattern stays at 0.71–0.72, indicating that their noise patterns do not get closer to the real hardware. SynQBench reaches r=0.92r=0.92 by contrast. Its MAE (0.50 pp) is still 2.2–2.3× lower than that of the calibrated baselines, even though each baseline's parameter was fitted to minimize exactly this metric.
  • Correlations. The residuals of the calibrated baselines remain at least 3.1× (spatial) and 5.1× (temporal) above SynQBench's. For SI1000-cal, the temporal residual even grows (4.29 → 9.78×10⁻³): scaling every channel by one factor also inflates its measurement and reset errors.

(b) Downstream training and evaluation

Training:

  • Best decoder performance. Training on SynQBench lowers Willow LER by 0.39–0.75 pp (12.6–18.7%) relative to Uniform-cal and SI1000-cal. The improvement holds for every architecture, and all 8 comparisons are significant.
  • Checkpoint: SynQBench is better and faster than the baselines' best. Even when each baseline is credited with its best epoch on the held-out shots, an oracle choice, the synthetically selected SynQBench checkpoint is better in all 20 comparisons, and reaches the calibrated baselines' best LER in 1.4–7.7× fewer epochs.

Evaluation: As evaluation benchmarks, the six distributions compare as follows:

  • Gap: mean |synthetic test LER − Willow LER| over 24 decoders.
  • MWPM gap: the same quantity for the fixed MWPM decoder.
  • Spearman$^*$: agreement of each benchmark's ranking with the Willow ranking of the 18 Transformer, GNN, and CNN decoders, whose LERs are closely spaced.
  • Brackets are 95% paired-shot bootstrap intervals.

* A benchmark is useful for evaluation only if it ranks decoders as the hardware does; otherwise it can favor the wrong architecture or training distribution even when its average LER is accurate. We add this rank agreement (Spearman correlation) in the revision.

BenchmarkGap (pp)MWPM gap (pp)Spearman
Uniform-base3.65 [3.38, 3.93]3.980.69 [0.59, 0.76]
SI1000-base3.72 [3.45, 4.00]4.080.46 [0.44, 0.57]
Uniform-cal0.75 [0.54, 1.02]0.820.69 [0.60, 0.77]
SI1000-cal1.99 [1.72, 2.27]2.260.49 [0.46, 0.60]
SynQBench-base1.40 [1.13, 1.68]1.670.86 [0.81, 0.92]
SynQBench0.23 [0.18, 0.41]0.040.97 [0.91, 0.99]

The downstream results mirror the statistics.

  • Gap. Calibration narrows the gap of each baseline (3.65 → 0.75 pp and 3.72 → 1.99 pp), but SynQBench remains the closest benchmark for 22 of the 24 decoders.
  • Ranking. Calibration leaves the ranking unchanged (Spearman 0.69 → 0.69 for Uniform, 0.46 → 0.49 for SI1000), mirroring the unchanged rate pattern, whereas SynQBench reaches 0.97. Pairwise ranking by SynQBench is 94.7% consistent with that of Willow, against only 75.7% for Uniform-cal and 65.8% for SI1000-cal.

Matching the overall noise scale is therefore not enough to train the best decoders or to rank them faithfully; the location-specific structure that SynQBench models is what carries the advantage.

AuthorsOpenReview ↗

General Response Part Three

3. Shot-level analyses: SynQBench-trained decoders correct more hardware shots, avoid more easy errors, and give more reliable confidence

Our initial submission evaluated training utility through a single aggregate metric per decoder, its hardware LER. Leveraging per-shot predictions across the 10,000 held-out shots, we now provide an in-depth analysis of how training on SynQBench changes decoding behavior on real hardware data. These analyses provide additional, shot-level evidence of SynQBench's training value, supporting Reviewer zTqS's Q3, Reviewer DgXc's Q2 and Q4, and Reviewer JMdf's concern about isolating the benefit over calibrated baselines.

(a) Paired shot outcomes: SynQBench-trained decoders correct more hardware shots. For each pair of decoders, won counts the shots that only the SynQBench-trained decoder decodes correctly, and lost counts the shots that only the other decoder decodes correctly. Δ\Delta = won − lost. When the two decoders disagree, the SynQBench-trained decoder is right 1.5–1.9 times as often as the calibrated-baseline decoder.

Architecturevs Uniform-cal (won / lost / Δ\Delta)vs SI1000-cal (won / lost / Δ\Delta)vs SynQBench-base (won / lost / Δ\Delta)
Transformer126 / 65 / +61141 / 81 / +6090 / 72 / +18
GNN123 / 73 / +50123 / 67 / +56100 / 66 / +34
CNN112 / 73 / +39132 / 71 / +6186 / 79 / +7
MLP182 / 107 / +75172 / 101 / +71126 / 105 / +21

(b) LER by shot difficulty: SynQBench wins at every difficulty level, with the largest relative gain on easy shots. We rank shots by how many of seven reference decoders fail on them: MWPM with two priors, plus the five decoders released with the Willow data. Each cell gives the LER and, in parentheses, the number of failed shots, both averaged over the four architectures.

Reference decoders failingShotsSynQBenchUniform-calSI1000-calSynQBench-base
0 of 79,3190.43% (40)0.66% (62)0.65% (61)0.52% (49)
1–2 of 735714.2% (51)19.9% (71)20.3% (73)16.4% (59)
≥ 3 of 732461.4% (199)65.7% (213)67.4% (219)62.4% (202)
  • On shots that every reference decoder solves, SynQBench-trained decoders make 34–36% fewer errors than the calibrated baselines; on moderately difficult shots, the reduction remains large: 29–30%.
  • On the hardest shots, the reduction is smaller (7–9%); these errors stem mainly from decoder capability and syndrome ambiguity, which better training data can only partly reduce.
  • Training on a mismatched distribution therefore adds errors mainly on shots that are not intrinsically hard, i.e., shots that most reference decoders solve.

(c) Per-stabilizer comparison: the gain covers the whole device. SynQBench wins the majority of disagreements on every one of the 24 stabilizers: win share 0.58–0.70 against the calibrated baselines and 0.51–0.60 against SynQBench-base. The advantage does not come from a few qubits.

(d) Prediction confidence: SynQBench-trained decoders give more reliable probabilities. Log-loss scores the probability a decoder assigns to the true logical outcome. Lower is better. Retained LER is the LER on each decoder's 90% most confident shots (as in post-selection).

ArchitectureLog-loss, Uniform-calLog-loss, SI1000-calLog-loss, SynQBenchRetained LER, Uniform-calRetained LER, SI1000-calRetained LER, SynQBench
Transformer0.0820.0850.0680.60%0.66%0.47%
GNN0.0840.0870.0720.69%0.68%0.48%
CNN0.0810.0840.0710.68%0.67%0.48%
MLP0.1140.1200.0981.29%1.36%0.98%
  • Log-loss is 12.9–19.9% lower than for the calibrated baselines. It is lower in all 20 comparisons, with paired 95% intervals excluding zero. The advantage persists (12.0–16.2%) after each decoder's overall confidence is rescaled on held-out folds, so it does not stem from global over- or under-confidence.
  • Retained LER is 22–31% lower than for the calibrated baselines; SynQBench retains the lowest LER for every architecture, baseline, and coverage level we tested.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.