acceptodds
Under review as a conference paper at ICLR 2027

Beyond Error Rates: Separating Difficulty from Decision-Rule Shift in Neural Quantum Error Correction

Abstract

Quantum error correction (QEC) uses measured error signatures to infer and correct physical faults. Neural decoders learn a syndrome-to-prediction mapping from samples generated under one noise model, but real quantum hardware does not operate under a fixed noise distribution. Existing evaluations typically report how the logical error rate changes with noise, but this cannot tell whether the problem has become harder or the optimal decoding decisions have changed. We introduce NSBench, a circuit-level benchmark that evaluates neural QEC decoders on shared syndrome streams across 18 streaming scenarios, a 75-configuration noise grid, and mechanism-specific strength sweeps. In tractable settings, exact source- and target-optimal references separate target difficulty from the cost of retaining old decoding decisions, which we call the shift cost. Broad multiplier-grid perturbations substantially increase the logical error rate of seven trained neural decoders at distance three and two released pre-decoders. In the shallow exact settings, the grid's shift cost remains small relative to target difficulty, whereas sufficiently strong local hotspots and correlated pairs have much larger relative shift costs. In absolute terms, the hardest grid targets can cost as much as structured faults. Adaptation recovers part of the shift cost when syndrome history reveals the change, but can also degrade performance when little shift exists. An exact Bayesian filter further shows that syndrome history alone cannot identify some noise changes. The logical error rate alone is therefore insufficient to determine the need for adaptation, and neural QEC decoders should be evaluated by how physical noise changes the decoding problem itself.

Primary area:
datasets and benchmarks
Keywords:
Quantum Error Correction, Neural Decoders, Distribution Shift, Adaptive Decoding, Test-Time Adaptation, Robustness Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gEuCOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies neural quantum error correction (QEC) decoders under changes in the physical noise distribution. The central observation is that an increase in logical error rate (LER) can arise for two different reasons: the target decoding problem may become intrinsically harder, or the noise change may alter the optimal syndrome-to-prediction decision rule. The authors distinguish these effects using target Bayes risk and a “shift cost,” which measures the additional target error incurred by retaining the source-optimal rule. They introduce NSBench, a circuit-level evaluation framework containing 18 streaming scenarios, a 75-configuration noise grid, and mechanism-specific strength sweeps, and use exact source- and target-optimal references where computationally feasible. Their experiments suggest that broad noise-rate changes can substantially increase LER while causing relatively little decision-rule shift, whereas sufficiently strong structured faults, such as local hotspots and correlated errors, can produce much larger shifts. The paper further evaluates adaptive neural decoders and finds that adaptation can recover some shift-related error when the change is observable from syndrome history, but can also degrade performance when little decision shift is present.

Critical strengths

Careful and well-designed empirical evaluation. The paper goes beyond simply comparing decoder LERs under different noise levels. A particularly intersting aspect is the use of exact source- and target-optimal decoding references in tractable settings. This allows the authors to determine whether performance degradation comes from increased intrinsic difficulty or from an actual change in the optimal decisions.

Critical weaknesses

Limited conceptual novelty. The central message, that changes in physical noise can either make decoding harder or change the optimal decoding rule, is useful but arguably intuitive to QEC experts. The paper itself acknowledges that its shift cost is essentially target regret and that the novelty lies more in the benchmark than in a new theoretical quantity.

Limited new ML methodology or theory. The paper is primarily an evaluation/benchmark study. It does not introduce a fundamentally new learning algorithm or substantial theoretical result. The error decomposition is useful diagnostically but mathematically straightforward, and the main adaptive method uses an existing FiLM mechanism.

Main evidence comes from synthetic noise. Most of the important results rely on controlled Stim simulations. The hotspots and correlated faults are stress-test mechanisms rather than calibrated models of a particular quantum device. Thus, it remains unclear how significant these decision shifts are under realistic hardware noise.

Exact results are limited to relatively small QEC settings. The strongest conclusions rely on cases where source- and target-optimal decisions can be computed exactly. Evidence becomes weaker for larger codes and deeper circuits, making scalability and practical relevance uncertain.

Questions for authors
  1. What is the principal novelty intended for the ML community? I found the benchmark and experiments carefully executed, but I am less convinced by the novelty of the underlying conceptual distinction. Separating intrinsic target difficulty from mismatch with a source-trained decision rule seems closely related to standard ideas in statistical decision theory and distribution shift. The paper would benefit from articulating more clearly what an ML researcher should learn that is not already implied by existing theory.

  2. The most interesting direction may be observability rather than shift cost itself. One of the paper's more compelling observations is that the existence of decision-rule shift does not imply that an adaptive decoder can recover it. In the aligned-pair example, different physical error explanations can generate the same syndrome while having opposite logical effects. Consequently, syndrome history may be insufficient to identify the relevant noise change. I found this issue more conceptually interesting than the basic difficulty-versus-shift decomposition. It would strengthen the paper considerably if the authors could provide a more general characterization of when decision-rule shifts are identifiable from unlabeled syndrome streams.

  3. Can shift cost be estimated in deployment? The benchmark can calculate shift cost because the source and target noise models are known in the controlled experiments. In actual deployment, however, the target model is generally unknown and logical labels are unavailable. It is therefore unclear how a system would determine that it is in a high-shift-cost regime and should activate adaptation. A practical estimator or even bounds based only on observable syndrome statistics would make the framework substantially more actionable.

  4. The adaptation results deserve more cautious interpretation. The finding that adaptation often degrades performance in small-shift scenarios is useful. The paper reports that the history adapter worsens 97 of 98 small-cost scenario–implementation cases, with 79 statistically significant regressions. However, it is difficult to determine how much of this is a fundamental consequence of unnecessary adaptation and how much is specific to the chosen FiLM adapter and its calibration. Experiments with a broader set of modern adaptation mechanisms would help separate these possibilities.

  5. I would like stronger evidence that the key observations are not simply expected consequences of known QEC behavior. The quantitative differences are interesting, for example, the large contrast between broad noise multipliers and structured faults, but the qualitative explanation is straightforward: structured perturbations alter the relative probabilities of competing physical explanations for a syndrome and can therefore change the maximum-likelihood decision. It would help if the authors identified observations that were genuinely unexpected before running the benchmark, or cases in which existing QEC intuition would have predicted the wrong outcome.

3 comments
AuthorsOpenReview ↗

Response to Reviewer gEuC (1/2)

We sincerely thank the reviewer for the careful review and the insightful comments. We respond to each weakness and question, in two comments.

"Limited conceptual novelty. The central message, that changes in physical noise can either make decoding harder or change the optimal decoding rule, is useful but arguably intuitive to QEC experts." / "Limited new ML methodology or theory."

"What is the principal novelty intended for the ML community?"

Thanks for your concern. The contribution is two-fold, and neither part is implied by the intuition or by existing theory. NSBench is a Datasets and Benchmarks track submission.

(1) A benchmark with exact references on both sides of a shift. In typical ML benchmarks test-time adaptation is scored without knowing the Bayes risk on either side of the shift. In QEC the circuit gives the exact joint distribution of syndrome and label, and in its tractable settings NSBench computes both optimal rules on the shots that every decoder sees, which turns the standard decomposition (Section 3.2) into a measurement. To the best of our knowledge, no existing benchmark for neural QEC decoders provides these two references. Its theory is the error identity of Appendix E and the identifiability rule of Appendix W.1.

(2) Findings that the intuition does not supply. The intuition says that both effects exist. It does not say which one the field's shifts produce, or how much of it unlabeled data can reveal.

  1. The multiplier grid is mostly a difficulty shift. A grid up to three, as used in recent out-of-distribution tests, raises error rates several times over, yet keeping the source rule costs at most 3.8% of target Bayes risk in the dense exact settings. A decoder can look robust or adaptive on that grid without its decision rule being tested.

  2. Part of the shift cost is invisible to unlabeled data. For the aligned pair, any rule that sees only syndromes loses at least 70% of the shift cost in one of two indistinguishable worlds (next answer).

  3. Adaptation has a measurable price where little is recoverable. A learned adapter loses accuracy on small-cost scenarios, including the no-shift control, and the rise in error rate does not predict where (Appendix W.4).

"It would strengthen the paper considerably if the authors could provide a more general characterization of when decision-rule shifts are identifiable from unlabeled syndrome streams."

We sincerely thank the reviewer for this valuable suggestion. The syndrome distribution fixes the firing rate of each detector footprint (the set of detectors a fault flips), but not how that rate splits between faults that flip the logical observable and faults that do not (Wagner et al., 2022, 2023, Blume-Kohout and Young, 2025, Zheng et al., 2026). Applied to decoding under shift, this gives three cases, stated as a rule in the revision (Section 5.4, Appendix W.1).

  • Recoverable in principle. A static shift that only reweights mechanisms of the source model.

  • Recoverable in our scenarios. A shift that adds a footprint whose syndrome the source rule already decodes with the correct logical value.

  • Hidden. A shift whose new fault produces a syndrome that the source rule decodes with the opposite logical value.

Scenario (d = 3, r = 2)Hidden from syndromesMeasured
Static hotspotNoExact Bayesian filter identifies it within about 50 shots. Adapter recovers 34 to 69%.
Moving hotspotNo, segment by segmentFilter recovers 48%. Adapter recovers nothing.
Correlated pairNoLabel-free estimate equals the exact cost.
Aligned pairYesLabel-free estimate reports 1.6% of the true cost.

The bound holds for every label-free method without a prior separating the two worlds. A single-qubit X error at the same probability on a different qubit produces exactly the same syndrome distribution as the aligned pair, so any syndrome-only rule loses at least 70% of the aligned pair's shift cost in one of the two. A census confirms the rule without exception. Among all 129 X faults on one, two or three data qubits at d = 3, r = 2, no single-qubit fault is hidden, and 18 of the 36 pairs and 48 of the 84 triples are. The ambiguity is not special to d = 3. An exact census at d = 5, r = 1 finds no hidden fault among the 325 X faults of weight one or two, and 292 of the 2,300 of weight three, exactly the set that the standard correctability condition predicts. One has the same syndrome as a weight-two fault with the opposite logical value, with equal label-free estimates (2.13e-3) and exact costs of 2.13e-3 and 19.65e-3.

References

  • Wagner, Kampermann, Bruß and Kliesch. Quantum 6, 809 (2022).

  • Wagner, Kampermann, Bruß and Kliesch. Physical Review Letters 130, 200601 (2023).

  • Blume-Kohout and Young. arXiv:2504.14643 (2025).

  • Zheng, Chu, Chen, Manes, Lee, Zhou and Jiang. arXiv:2601.22286 (2026).

AuthorsOpenReview ↗

Response to Reviewer gEuC (2/2)

(Continued, part 2 of 2.)

"Can shift cost be estimated in deployment? ... A practical estimator or even bounds based only on observable syndrome statistics would make the framework substantially more actionable."

Thanks for your question. The revision gives both, the estimator and the bound (Appendix W.1).

  • Estimator. When a shift only reweights mechanisms of the source model, the population shift cost is computed from syndrome statistics and the source model, with no labels. A finite-sample version is close to the exact value on static shifts with about half a million unlabeled shots.

  • Bound. Syndromes cannot certify that a shift belongs to this class. The aligned pair and its twin have the same syndrome distribution and shift costs of 18.68e-3 and 0.30e-3, so no label-free monitor is correct for every shift.

  • Scope. The estimator needs far more than 512 shots and is biased on time-varying noise. The paper uses it as an audit of which scenarios are recoverable in principle, not as a deployment monitor.

"it is difficult to determine how much of this is a fundamental consequence of unnecessary adaptation and how much is specific to the chosen FiLM adapter and its calibration."

The revision separates the two against an exact reference.

  • A hedging cost that exact inference also pays. The exact Bayesian filter under the same prior also loses on the small-cost scenarios, by a mean of 0.18e-3 on the 11 scenarios where its prior applies, none of them individually significant.

  • Adapter-specific part. The learned adapter pays about three times that, relative to its own frozen backbone. It regresses significantly on 78 of the 98 small-cost cells (exact test, the submitted Monte Carlo count was 79).

  • Gating and joint training. A delayed gate lowers the count to 44 of 98 (Table W.4, 57 to 27 on the 70 in-house cells of Section 5.4), and joint training with the measurement multiplier to 68 (not a significant improvement). A gate driven by the exact filter removes every significant regression, but it opens on 1.3% of those shots and gives up the static-hotspot recovery.

  • Other methods. TENT (Wang et al., 2021) and four of the six matching-based tiers show no significant regression.

All learned variants in the revision keep the FiLM form, so the regression count is a statement about this adapter family (Section 6.3).

"It would help if the authors identified observations that were genuinely unexpected before running the benchmark, or cases in which existing QEC intuition would have predicted the wrong outcome."

Thank you for this suggestion. Three results run against the intuition that a structured fault changes the decision as soon as it reweights competing explanations.

  1. A fault that mimics a likelier error shifts the rule later. The aligned pair crosses the 4% level at a higher strength than the horizontal pair (0.012 against 0.0075 at d = 3, r = 2), because its decision flips only once the pair is more likely than the single error it mimics. The submission had recorded the opposite prediction before the run (Appendix G).

  2. More changed decisions do not mean more cost. At the benchmark strengths the grid changes the optimal decision on more probability mass (0.62 to 0.73%) than the horizontal pair does (0.37%), and it costs less than half as much, because the grid flips decisions that were near ties.

  3. Rate changes cost little when all rates scale together, and more when only some do. Scaling every rate by the same factor costs at most 1.3% of target Bayes risk up to a multiplier of ten. Scaling only some axes crosses 4% at a multiplier of four (d = 3, r = 2, exact). The benchmark grid stays cheap because it stops at three.

"Main evidence comes from synthetic noise." / "Exact results are limited to relatively small QEC settings."

An exact optimal rule on both sides of a shift exists only for a specified noise model, so the hotspots and pairs are stress tests by design (Section 6.3). Section 5.2 and Appendix U compare the hotspot strengths with published single-qubit relaxation fluctuations, and the paper also analyses 420 recorded Willow configurations (Appendix S).

The central contrast is measured with exact references, not extrapolated. At d = 3, r = 8 and d = 5, r = 5 the largest evaluated grid point costs 4.4% and 5.8% of target Bayes risk, while the structural scenarios cost 19 to 173% (Table 1). At d = 7, r = 2 the revision resolves the aligned pair at 124% (95% interval 98 to 158%), against 10.4% for the largest grid point. The other structural cases at d = 7 remain unresolved.

We hope these responses address the reviewer's concerns, and we would be glad to clarify any remaining point during the discussion period.

Reference

  • Wang, Shelhamer, Liu, Olshausen and Darrell. Tent: Fully test-time adaptation by entropy minimization. ICLR (2021).
AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer gEuC,

We sincerely appreciate your thoughtful review, particularly your suggestion on observability, which helped us strengthen the paper.

As the discussion period approaches its end, we would be grateful for your feedback on our response and revision. If any concerns remain, we would be happy to provide further clarification.

Thank you again for your time and consideration.

Reviewer_xSCBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces NSBench, a circuit-level benchmark for evaluating neural quantum error correction (QEC) decoders under changing physical noise. Unlike conventional evaluations based solely on logical error rate (LER), NSBench distinguishes increased intrinsic decoding difficulty from changes in the optimal decision rule using exact source- and target-optimal references. The benchmark includes 18 streaming scenarios, 75 noise configurations, and mechanism-specific strength sweeps, evaluating seven trained neural decoders, two released pre-decoders, and several classical and adaptive methods. Experiments show that broad noise perturbations can increase LER without substantially changing optimal decisions, whereas strong localized or correlated faults induce significant decision shifts. Furthermore, history-based adaptation can recover part of the shift-related error but may degrade performance when little shift exists, while Bayesian filtering reveals fundamental limitations in identifying noise changes from syndrome observations alone.

Critical strengths
  1. Well-motivated problem formulation and a meaningful evaluation perspective.

The paper addresses an important limitation in existing neural QEC evaluations: an increase in logical error rate does not necessarily imply that the decoder's learned decision rule has become inappropriate. By explicitly distinguishing target Bayes risk from the regret of retaining the source-optimal decoder, the proposed framework provides a more informative interpretation of performance degradation under noise shifts. This distinction is particularly relevant for evaluating adaptive decoders, since additional adaptation is only potentially useful when the original decoding decisions have become suboptimal. Although the underlying regret formulation has precedents in decoder-mismatch studies, its systematic application to neural QEC robustness evaluation is valuable.

  1. Comprehensive and carefully controlled benchmark design.

NSBench covers diverse noise mechanisms, including global rate changes, temporal drift, localized hotspots, correlated errors, and noise-composition shifts. The combination of streaming scenarios, static noise grids, and mechanism-specific strength sweeps enables a more detailed characterization of robustness than evaluations based on a small set of noise levels. The shared-syndrome-stream protocol, causal evaluation constraints, episode-level statistical analysis, and explicit frozen-decoder baselines further strengthen experimental consistency and reproducibility. These design choices make the benchmark useful for separating the effects of noise characteristics from differences between decoder implementations.

  1. Rigorous use of exact decoding references to support the central analysis.

A notable technical strength is the use of exact source- and target-optimal decoding decisions rather than relying exclusively on approximate classical decoders as references. The paper combines dense enumeration for small circuits with exact per-shot posterior inference for deeper circuits, and provides consistency checks between these approaches. This allows the authors to quantify decision-rule mismatch directly and distinguish it from decoder-specific approximation errors. The decomposition of excess decoding error into shift cost and a signed decoder-specific residual also helps avoid incorrectly attributing all performance degradation to distribution shift.

Critical weaknesses
  1. Limited conceptual novelty compared with prior decoder-mismatch studies

While the proposed NSBench provides a systematic evaluation framework, its central distinction between target difficulty and decision-rule shift is closely related to existing research on decoder mismatch, particularly Darmawan (2025), which already investigates the performance loss caused by using an incorrect noise model and the benefits of adapting to local noise. The shift cost introduced in this paper is essentially the target regret of the source-optimal decoder, rather than a fundamentally new theoretical formulation. Therefore, it remains unclear how much of the paper's central findings represent genuinely new insights beyond existing noise-mismatch analyses.

  1. Limited scalability and practical validation of the central findings

Although the paper presents extensive experiments, its strongest conclusions rely primarily on small surface-code settings, particularly (d=3), where exact optimal decoding references are computationally feasible. The evidence becomes considerably weaker at larger code distances: some deeper-circuit results rely on very limited samples, several structured shifts remain statistically unresolved at (d=7), and the matching-based surrogate can substantially underestimate the shift cost of certain noise-grid perturbations. Moreover, the benchmark primarily uses synthetic noise models, while the hardware-data analysis does not directly validate the proposed difficulty–shift decomposition. These limitations raise concerns about whether the observed differences between broad noise perturbations and structured faults generalize to practically relevant QEC systems.

  1. Insufficient evidence that the observed adaptation failures generalize beyond the specific adapter design and training distribution

One of the paper's central findings is that adaptation frequently degrades performance when the shift cost is small, with regressions observed in 97 of 98 small-cost scenario–implementation pairs. However, these results are primarily obtained using a particular FiLM-based history adapter trained on a restricted noise family that does not vary the measurement-error multiplier. The paper's own ablations show that changing the training distribution or delaying adapter activation can substantially reduce regressions, suggesting that the observed failures may partly result from training-distribution mismatch or adapter calibration rather than an inherent limitation of adaptation under small decision shifts.

Questions for authors
  1. Could the authors provide a more direct comparison with Darmawan (2025), ideally by evaluating comparable noise-shift settings and demonstrating which conclusions or capabilities are uniquely enabled by NSBench? Such an analysis would help clarify whether the primary contribution lies in new scientific findings or mainly in a more comprehensive benchmark implementation.

  2. Could the authors provide additional statistically reliable experiments at larger code distances, and evaluate the proposed conclusions under more realistic or hardware-calibrated noise models? Even a targeted study with well-characterized uncertainty would significantly strengthen the practical relevance of the findings.

  3. Could the authors evaluate stronger or better-calibrated adaptation baselines, such as jointly trained adapters covering both structured and measurement-noise shifts, or confidence-gated adaptation mechanisms that selectively activate updates? Comparing these approaches under matched and unseen noise conditions would help determine whether the reported negative transfer reflects a general challenge for adaptive QEC decoders or a limitation of the specific implementation.

4 comments
AuthorsOpenReview ↗

Response to Reviewer xSCB (1/2)

We thank the reviewer for the careful review and for the assessment of the benchmark design and the exact references. Each weakness matches one question, so we answer them together, in two comments.

"Limited conceptual novelty compared with prior decoder-mismatch studies"

"Could the authors provide a more direct comparison with Darmawan (2025), ideally by evaluating comparable noise-shift settings and demonstrating which conclusions or capabilities are uniquely enabled by NSBench?"

NSBench contributes new findings, not only a broader implementation. The two works share the quantity, the loss from decoding with a mischaracterized noise model, and Appendix C credits that work for it. The revision adds a comparison of scope to Appendix C and states the relation in the introduction. The comparison is one of scope, not of numbers.

Darmawan (2025)NSBench
NoiseMainly static and uncorrelated. Includes inhomogeneous noise, which overlaps with our hotspot, and coherent noise.Static and time-varying. Hotspots, correlated faults and drift. No coherent noise.
DecodersTensor-network decoding, with a matching baselineLearned decoders, released pre-decoders and adaptive methods
Visibility of a mismatch in syndromesNot studiedMeasured, with a two-model bound that holds for every label-free rule

The conclusions that NSBench uniquely enables are these: the multiplier grid that the field uses as its out-of-distribution test is mostly a difficulty shift, the evaluated learned adapter loses accuracy where little is recoverable, and part of a shift cannot be recovered from unlabeled data at all.

"Limited scalability and practical validation of the central findings"

"Could the authors provide additional statistically reliable experiments at larger code distances ..."

Thanks for your concern. The revision adds the targeted study the reviewer asks for, at d = 7 with measured interval coverage. The largest evaluated grid point costs 10.4% of its Bayes risk, with a shot-level 95% interval of 7.3 to 13.9% whose measured coverage is 95%. The aligned pair at its benchmark strength costs 124% (95% interval 98 to 158%), so these two cases are separated at d = 7 as well.

Through d = 5, in the submission. At d = 3, r = 8 and at d = 5, r = 5 the largest evaluated grid point costs 4.4% and 5.8% of its Bayes risk, and the structural scenarios cost 19 to 173%. The contrast between the grid and the structural faults therefore holds with exact references through d = 5.

New at d = 7, r = 2. Six blocks of 512 shots were added to each of four grid cases, from 1,024 to 4,096 shots, with seeds, shot count and analysis fixed before decoding. The largest grid point (gate, idling and Z-bias axes at a multiplier of three) costs 10.4%, above the 3.8% maximum of the dense settings, and the other three cost 1.5 to 5.2%. The revision reports the measured coverage of the Table 1 intervals that we could check and adds shot-level intervals (Appendix W.3). The aligned-pair estimate uses 33,280 shots, stratified by the pair's fault events and reweighted exactly, with a preregistered analysis, and its interval has a measured coverage of 94.5% (Appendix H). The other structural cases at d = 7 remain unresolved.

Surrogate and 25-round rows. No conclusion above uses the matching-based surrogate, which the paper uses only to follow the structural sweep to larger distance. The 25-round rows of Table 1 rest on one 512-shot block per case and are reported as preliminary point estimates (Section 5.5, Appendix U).

Reference

  • Darmawan. Optimal adaptation of surface-code decoders to local noise. Physical Review A 112, 042431 (2025). arXiv:2403.08706.
AuthorsOpenReview ↗

Response to Reviewer xSCB (2/2)

(Continued, part 2 of 2.)

"... and evaluate the proposed conclusions under more realistic or hardware-calibrated noise models?"

Thanks for your question. The revision adds a measurement on the public Willow memory data (Google Quantum AI, 2025, Appendix S).

What was measured. We decoded the recorded syndromes exactly under two released noise models and scored both rules against the recorded logical outcomes. One model is generic. The other is a prior that the hardware team tuned for decoding accuracy on a calibration run.

Result. At d = 3 with 10 or more rounds the tuned-prior rule is better on all 9 patches, with a median reduction in logical error rate of 2.4% (patch bootstrap 95% interval 1.6 to 4.3%). Without the 13-round circuits, which match the calibration run, the result holds on 8 of 9 patches. At one round, and at d = 5 where only one-round circuits are within reach of the exact decoder, no difference is resolved.

Scope. The benchmark results are simulated by design, because an exact optimal rule on both sides of a shift requires a specified noise model. The Willow measurement is a difference between two decoding rules on recorded shots. It is not a shift cost and not a validation of the decomposition, because the true device noise is unknown and no exact reference exists for it. The patch-level test was specified after a first, per-round analysis that did not meet its written criteria, and Appendix S reports this in full.

"Insufficient evidence that the observed adaptation failures generalize beyond the specific adapter design and training distribution"

"Could the authors evaluate stronger or better-calibrated adaptation baselines, such as jointly trained adapters covering both structured and measurement-noise shifts, or confidence-gated adaptation mechanisms that selectively activate updates?"

We sincerely thank you for the concrete suggestions. The revision adds a jointly trained adapter, gates that activate the adapter selectively, and an exact reference for them. The negative transfer has a general component and an implementation component, and the revision measures each.

VariantSignificant regressions on the 98 small-cost cellsNote
Submitted adapter78Exact test. The submitted Monte Carlo count was 79.
Delayed gate44Table W.4. On the 70 in-house cells the count falls from 57 to 27 (Section 5.4).
Jointly trained, with the measurement multiplier added68Not a significant improvement in a paired comparison.
Gate driven by the exact filternoneGives up the static-hotspot recovery.
  • General component: the price of hedging. The exact Bayesian filter under the same prior also loses on these scenarios, by a mean of 0.18e-3 on the 11 scenarios where its prior applies, none of them individually significant. This cost does not depend on any adapter.

  • Implementation component. The learned adapter pays about three times that, relative to its own frozen backbone. On the measurement-dominated scenario the jointly trained adapter lowers the mean regression from 1.28e-3 to 0.76e-3, and all 7 cells there still regress significantly. The missing training axis therefore explains part of the harm, not all of it.

  • Seen and unseen measurement axis. The submitted adapter regresses significantly on 57 of the 77 cells whose scenarios keep the measurement multiplier of its training family, and on all 21 cells of the three scenarios that vary it (Appendix L).

  • Other adaptive methods. TENT (Wang et al., 2021) and four of the six matching-based tiers show no significant regression.

  • Scope. The exact-filter gate is a reference and not a learned confidence gate, and all learned variants in the revision keep the FiLM form. The regression count is therefore a statement about this adapter family, as Section 6.3 of the submission states.

Among these variants, only the exact-filter gate removes every significant regression, and it gives up the static-hotspot recovery. Both sides of this trade are measured on the same shots against an exact reference.

We hope these responses address the reviewer's concerns, and we would be glad to clarify any remaining point during the discussion period.

References

  • Wang, Shelhamer, Liu, Olshausen and Darrell. Tent: Fully test-time adaptation by entropy minimization. ICLR (2021).

  • Google Quantum AI and Collaborators. Quantum error correction below the surface code threshold. Nature 638, 920-926 (2025).

Reviewer_xSCBOpenReview ↗

Reply to rebuttal

Thank you for your detailed responses and clarifications. I appreciate the authors' efforts in addressing my concerns, and I acknowledge that most of my questions have been satisfactorily resolved. However, considering the remaining concerns regarding the novelty of the work as other reviewers also pointed out and its overall contribution, I have decided to maintain my initial positive score of 3.

AuthorsOpenReview ↗

Thank You for Your Constructive Feedback

Dear Reviewer xSCB,

Thank you very much for your thoughtful follow-up. We are pleased that most of your questions have been satisfactorily resolved.

We sincerely appreciate your constructive feedback, which has helped us strengthen the paper and clarify its contributions.

Thank you again for your time and consideration.

Reviewer_zPwMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a circuit-level benchmark that evaluates neural QEC decoders on shared syndrome streams across 18 streaming scenarios, a 75-configuration noise grid, and mechanism-specific strength sweeps. This benchmark is designed to study trained neural decoders, released pre-decoders, and adaptive methods.

Critical strengths

The paper introduces a circuit-level benchmark that measures both decoding difficulty and decision shift separately, on the same syndrome streams. The paper proved that a rise in logical error rate does not by itself predict decision shift. Across the settings with exact dense references, the evaluated noise grids have shift costs of at most 3.8% of target Bayes risk, while sufficiently strong local hotspots and correlated pairs reach 14–87%. Exact per-shot inference extends this separation through code distance five. The paper verifies that this distinction changes how decoder adaptation should be evaluated. Decision shift explains one component of frozen-decoder failure. In our experiments, a label free history adapter recovers 34–69% of the static-hotspot shift cost but significantly worsens 79 of 98 small-cost scenario–implementation pairs, by a mean of 0.64 Å~ 10−3 LER. An exact Bayesian filter also identifies shifts that syndrome history alone cannot resolve.

Critical weaknesses

The paper has done comprehensive experiments in terms of number of noise scenarios and number of decoders to evaluate results. How to prove the novelty, generalizability of the proposed benchmark and its main conclusions are not clear to me. The methodological contribution seems very limited. It would be helpful for the authors to clarify the methodological novelty of NSBench beyond combining existing noise models, decoding architectures and evaluation settings. And the theoretical analysis isn’t clear.

Questions for authors

The distinction between target difficulty and shift cost is an important perspective of the benchmark. The shift cost formulation looks like closely related to the notion of risk under distribution shift in classical ML. I am not very convinced about the novelty of this formulation relative to existing distribution shift and domain adaptation analyses.
The benchmark includes 75 noise configurations. Can you explain how these configurations were selected and how to categorize these configurations to 75 categories. How to prove they describe or cover realistic noise in quantum hardware. Are they collected from real quantum hardware or they are synthetic data? Could authors clarify if any of the experiments are tested on real quantum hardware? I would like to understand the adaptation effectiveness of generalizing to practical quantum hardware.

2 comments
AuthorsOpenReview ↗

Response to Reviewer zPwM

We thank the reviewer for the careful review. NSBench was submitted to the Datasets and Benchmarks track. Its primary contribution is a benchmark for evaluating neural QEC decoders under noise shift, rather than a new learning method. QEC needs such a benchmark because the noise of a device changes during operation while a neural decoder is trained under one noise model, and the error rate alone cannot show whether the decoder then needs adaptation.

"How to prove the novelty, generalizability of the proposed benchmark and its main conclusions are not clear to me. ... clarify the methodological novelty of NSBench beyond combining existing noise models, decoding architectures and evaluation settings. And the theoretical analysis isn't clear."

NSBench goes beyond combining existing evaluation settings in two ways.

(1) Exact measurement of decision-rule shift. In typical ML benchmarks the target-optimal rule is unknown, so the loss of keeping the old rule is bounded rather than measured. In QEC the circuit gives the exact joint distribution of syndrome and label, so NSBench computes both optimal rules exactly and scores every decoder against them on the same shots. To the best of our knowledge, NSBench is the first benchmark for neural QEC decoders that scores them against the exact optimal rule on both sides of a noise shift.

(2) Findings that change how decoders are evaluated. The multiplier grid that the field uses as its out-of-distribution test costs at most 3.8% of target Bayes risk in the dense exact settings, while the benchmark's hotspot and pair scenarios cost 14 to 87%. The evaluated learned adapter loses accuracy where little is recoverable, and part of a shift cannot be recovered from unlabeled data at all (Section 5.4, Appendix W.1).

Its theory is the error identity of Appendix E and the identifiability rule of Appendix W.1. The protocol applies to any code with a detector error model, with exact references wherever exact decoding is tractable. The numbers are specific to the surface-code memory settings we ran (Section 6.3).

"The shift cost formulation looks like closely related to the notion of risk under distribution shift in classical ML. I am not very convinced about the novelty of this formulation relative to existing distribution shift and domain adaptation analyses."

Thanks for your question. Shift cost is deliberately defined as standard regret, making it directly interpretable to ML readers (Section 3.2). Domain-adaptation bounds such as that of Ben-David et al. (2010) contain a joint-error term that is in general not identifiable from unlabeled target data. NSBench instead computes the regret exactly in its tractable simulated settings. At d = 3, r = 2 a global rate change by 1.5 and a threefold center-row hotspot give the source-optimal rule nearly the same error rate, 0.0228 and 0.0214, but shift costs of 0.01% and 9.3% of target Bayes risk (Section 5.2).

"Can you explain how these configurations were selected ... Are they collected from real quantum hardware or they are synthetic data?"

The 75 configurations form one grid: the 15 nonempty subsets of four noise axes (measurement, gate, idling, Z bias), each at five multipliers from 1.2 to 3 (Section 4.1, Appendix J). It extends the 11 multi-axis combinations of the QAdapt grid to all 15 subsets. The configurations are synthetic by design. For realism, the paper compares the hotspot strengths with published single-qubit relaxation fluctuations (Section 5.2, Appendix U) and analyses recorded hardware shots (next answer). Circuit-level simulation with specified noise is the standard setting for evaluating decoders in the QEC community, because only a specified noise model gives a known ground truth.

"Could authors clarify if any of the experiments are tested on real quantum hardware? I would like to understand the adaptation effectiveness of generalizing to practical quantum hardware."

Yes, on recorded hardware data. Decoding is classical post-processing, so a decoder is tested on hardware by decoding the syndromes a device recorded. The paper analyses 420 recorded Willow configurations of 50,000 shots each (Appendix S), where an adaptive matching decoder shows no aggregate gain from global rate tracking and small pooled gains from local reweighting. The revision adds a comparison of the exact decoding rules under two released noise models on 103 of them. Neural decoders are evaluated on the simulated tiers, where an exact reference exists to score them.

We hope these responses address the reviewer's concerns, and we would be glad to clarify any remaining point during the discussion period.

References

  • Ben-David, Blitzer, Crammer, Kulesza, Pereira and Wortman Vaughan. A theory of learning from different domains. Machine Learning 79, 151-175 (2010).
  • Miao, Luo, Shan and Sun. QAdapt. arXiv:2607.28422 (2026).
AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer zPwM,

We sincerely appreciate your careful review and thoughtful questions on novelty, benchmark design, and hardware evaluation.

As the discussion period approaches its end, we would be grateful for your feedback on our response and revision. If any concerns remain or further clarification would be helpful, please do not hesitate to let us know.

Thank you again for your time and consideration.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response: what NSBench measures and what the revision adds

We sincerely thank all three reviewers for their time and constructive feedback, and we are encouraged that they found the benchmark design careful and well controlled and the use of exact references rigorous. NSBench is a benchmark for neural QEC decoders under changing noise. In the tractable settings it computes the exact optimal rule before and after a shift and scores every decoder against both on the same shots. This separates what an error-rate comparison merges: how much harder the target became, how much the optimal decisions changed, and how much of that change a label-free method can recover.

The revised PDF strengthens this measurement in six ways. Changed text is marked in colour. The title and the abstract are unchanged.

  1. A rule for what unlabeled syndromes can and cannot recover. The submission showed for one case that syndrome history cannot identify some noise changes (Sections 5.4 and 6.2). The revision turns this into a rule, with a bound over two indistinguishable noise models that holds for every label-free method (Section 5.4, Appendix W.1). This responds to gEuC (observability, deployment).

  2. A precise statement about multiplier grids. At d = 3, r = 2 uniform rate scaling costs at most 1.3% of target Bayes risk up to a multiplier of ten. Non-uniform reweighting costs more, and it is now reported beyond the benchmark's cap of three (Section 5.2, Appendix W.2). This responds to gEuC (unexpected observations).

  3. The relation to prior work, stated in the main text. Appendix C of the submission credits decoder-mismatch work for the regret. The revised introduction states this relation and the relation to identifiability work, and Appendix C adds a comparison of scope with Darmawan (2025). This responds to xSCB, gEuC and zPwM on novelty.

  4. Stronger adaptation baselines with an exact reference. Gated and jointly trained variants and an exact-filter comparison are added (Sections 5.4 and 6.3, Appendix W.4). Adapter claims stay scoped to the evaluated adapter family (Section 6.3). This responds to xSCB and gEuC on adaptation.

  5. A further measurement on recorded hardware shots. Appendix S of the submission analysed recorded Willow shots, and the revision adds a comparison of two exact decoding rules on them. This responds to xSCB and zPwM on hardware.

  6. Resolved costs at d = 7. Four d = 7 grid cases are extended from 1,024 to 4,096 shots (Section 5.5, Table 1, Appendices H and W.3). The largest costs 10.4% of target Bayes risk (shot-level 95% interval 7.3 to 13.9%). The aligned pair at its benchmark strength is now resolved at d = 7 and costs 124% (95% interval 98 to 158%). The other structural cases at d = 7 remain unresolved. This responds to xSCB and gEuC on scale.

Corrections from our own audit of the submission

  • The count of significant regressions is 78 under an exact test, in place of 79 under a Monte Carlo test, as Appendix L of the submission anticipated.

  • With the larger d = 7 sample three grid values change (9.1 to 10.4%, 6.6 to 4.7%, 3.9 to 1.5%).

  • Section 5.4 no longer says that the aligned-pair ambiguity disappears at d = 5. Section 5.2 now says that structured faults cost more, not that they change more decisions.

  • Smaller corrections, including the measured coverage of the Table 1 intervals, are marked in colour (Sections 4.1, 5.2, 5.3 and 5.5, Appendices D, F, G, H, L, O, Q, R and W.3).

Reference

  • Darmawan. Optimal adaptation of surface-code decoders to local noise. Physical Review A 112, 042431 (2025). arXiv:2403.08706.
AuthorsOpenReview ↗

Summary to Reviewers, ACs and SACs

Dear Reviewers, ACs and SACs,

We sincerely thank the reviewers for their thoughtful feedback and constructive discussion. We summarize the core contributions of NSBench and the additional results provided during rebuttal.

Core contributions. NSBench is a Datasets and Benchmarks submission. To the best of our knowledge, it is the first benchmark to score neural QEC decoders against the exact optimal decoding rule on both sides of a noise shift, in settings where exact inference is tractable. This enables three findings that conventional logical-error-rate evaluations cannot establish:

  1. The evaluated multiplier grid is predominantly a difficulty shift, with shift costs of at most 3.8% of target Bayes risk in the dense exact settings, whereas sufficiently strong hotspots and correlated faults reach 14–87%.
  2. Certain decision-rule shifts cannot be identified from unlabeled syndromes alone, placing a limit on what label-free adaptation can recover.
  3. The evaluated learned adapter loses accuracy when little shift cost is recoverable, making the cost of unnecessary adaptation measurable.

Rebuttal contributions. In response to the reviewers' questions, we strengthened the evidence supporting these findings:

  • Novelty: We clarified the distinction between NSBench and prior decoder-mismatch and identifiability research in the introduction and Appendix C, including a scope comparison with Darmawan (2025).
  • Identifiability: We stated the observability criterion as a rule, established a bound for label-free methods under syndrome-indistinguishable noise models, and added a label-free shift-cost estimator (Section 5.4, Appendix W.1).
  • Adaptation: We evaluated jointly trained and gated adapters against an exact-filter reference (Sections 5.4 and 6.3, Appendix W.4).
  • Scalability and hardware: We resolved the aligned-pair shift cost at d = 7 (124%, 95% CI: 98–158%), compared with 10.4% for the largest evaluated grid point, and added a comparison of two exact decoding rules under released noise models on 103 recorded Willow configurations.

We sincerely thank the reviewers, ACs and SACs again for their time and consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.