- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies neural quantum error correction (QEC) decoders under changes in the physical noise distribution. The central observation is that an increase in logical error rate (LER) can arise for two different reasons: the target decoding problem may become intrinsically harder, or the noise change may alter the optimal syndrome-to-prediction decision rule. The authors distinguish these effects using target Bayes risk and a “shift cost,” which measures the additional target error incurred by retaining the source-optimal rule. They introduce NSBench, a circuit-level evaluation framework containing 18 streaming scenarios, a 75-configuration noise grid, and mechanism-specific strength sweeps, and use exact source- and target-optimal references where computationally feasible. Their experiments suggest that broad noise-rate changes can substantially increase LER while causing relatively little decision-rule shift, whereas sufficiently strong structured faults, such as local hotspots and correlated errors, can produce much larger shifts. The paper further evaluates adaptive neural decoders and finds that adaptation can recover some shift-related error when the change is observable from syndrome history, but can also degrade performance when little decision shift is present.
Critical strengths
Careful and well-designed empirical evaluation. The paper goes beyond simply comparing decoder LERs under different noise levels. A particularly intersting aspect is the use of exact source- and target-optimal decoding references in tractable settings. This allows the authors to determine whether performance degradation comes from increased intrinsic difficulty or from an actual change in the optimal decisions.
Critical weaknesses
Limited conceptual novelty. The central message, that changes in physical noise can either make decoding harder or change the optimal decoding rule, is useful but arguably intuitive to QEC experts. The paper itself acknowledges that its shift cost is essentially target regret and that the novelty lies more in the benchmark than in a new theoretical quantity.
Limited new ML methodology or theory. The paper is primarily an evaluation/benchmark study. It does not introduce a fundamentally new learning algorithm or substantial theoretical result. The error decomposition is useful diagnostically but mathematically straightforward, and the main adaptive method uses an existing FiLM mechanism.
Main evidence comes from synthetic noise. Most of the important results rely on controlled Stim simulations. The hotspots and correlated faults are stress-test mechanisms rather than calibrated models of a particular quantum device. Thus, it remains unclear how significant these decision shifts are under realistic hardware noise.
Exact results are limited to relatively small QEC settings. The strongest conclusions rely on cases where source- and target-optimal decisions can be computed exactly. Evidence becomes weaker for larger codes and deeper circuits, making scalability and practical relevance uncertain.
Questions for authors
-
What is the principal novelty intended for the ML community? I found the benchmark and experiments carefully executed, but I am less convinced by the novelty of the underlying conceptual distinction. Separating intrinsic target difficulty from mismatch with a source-trained decision rule seems closely related to standard ideas in statistical decision theory and distribution shift. The paper would benefit from articulating more clearly what an ML researcher should learn that is not already implied by existing theory.
-
The most interesting direction may be observability rather than shift cost itself. One of the paper's more compelling observations is that the existence of decision-rule shift does not imply that an adaptive decoder can recover it. In the aligned-pair example, different physical error explanations can generate the same syndrome while having opposite logical effects. Consequently, syndrome history may be insufficient to identify the relevant noise change. I found this issue more conceptually interesting than the basic difficulty-versus-shift decomposition. It would strengthen the paper considerably if the authors could provide a more general characterization of when decision-rule shifts are identifiable from unlabeled syndrome streams.
-
Can shift cost be estimated in deployment? The benchmark can calculate shift cost because the source and target noise models are known in the controlled experiments. In actual deployment, however, the target model is generally unknown and logical labels are unavailable. It is therefore unclear how a system would determine that it is in a high-shift-cost regime and should activate adaptation. A practical estimator or even bounds based only on observable syndrome statistics would make the framework substantially more actionable.
-
The adaptation results deserve more cautious interpretation. The finding that adaptation often degrades performance in small-shift scenarios is useful. The paper reports that the history adapter worsens 97 of 98 small-cost scenario–implementation cases, with 79 statistically significant regressions. However, it is difficult to determine how much of this is a fundamental consequence of unnecessary adaptation and how much is specific to the chosen FiLM adapter and its calibration. Experiments with a broader set of modern adaptation mechanisms would help separate these possibilities.
-
I would like stronger evidence that the key observations are not simply expected consequences of known QEC behavior. The quantitative differences are interesting, for example, the large contrast between broad noise multipliers and structured faults, but the qualitative explanation is straightforward: structured perturbations alter the relative probabilities of competing physical explanations for a syndrome and can therefore change the maximum-likelihood decision. It would help if the authors identified observations that were genuinely unexpected before running the benchmark, or cases in which existing QEC intuition would have predicted the wrong outcome.
3 comments
Response to Reviewer gEuC (2/2)
(Continued, part 2 of 2.)
"Can shift cost be estimated in deployment? ... A practical estimator or even bounds based only on observable syndrome statistics would make the framework substantially more actionable."
Thanks for your question. The revision gives both, the estimator and the bound (Appendix W.1).
-
Estimator. When a shift only reweights mechanisms of the source model, the population shift cost is computed from syndrome statistics and the source model, with no labels. A finite-sample version is close to the exact value on static shifts with about half a million unlabeled shots.
-
Bound. Syndromes cannot certify that a shift belongs to this class. The aligned pair and its twin have the same syndrome distribution and shift costs of 18.68e-3 and 0.30e-3, so no label-free monitor is correct for every shift.
-
Scope. The estimator needs far more than 512 shots and is biased on time-varying noise. The paper uses it as an audit of which scenarios are recoverable in principle, not as a deployment monitor.
"it is difficult to determine how much of this is a fundamental consequence of unnecessary adaptation and how much is specific to the chosen FiLM adapter and its calibration."
The revision separates the two against an exact reference.
-
A hedging cost that exact inference also pays. The exact Bayesian filter under the same prior also loses on the small-cost scenarios, by a mean of 0.18e-3 on the 11 scenarios where its prior applies, none of them individually significant.
-
Adapter-specific part. The learned adapter pays about three times that, relative to its own frozen backbone. It regresses significantly on 78 of the 98 small-cost cells (exact test, the submitted Monte Carlo count was 79).
-
Gating and joint training. A delayed gate lowers the count to 44 of 98 (Table W.4, 57 to 27 on the 70 in-house cells of Section 5.4), and joint training with the measurement multiplier to 68 (not a significant improvement). A gate driven by the exact filter removes every significant regression, but it opens on 1.3% of those shots and gives up the static-hotspot recovery.
-
Other methods. TENT (Wang et al., 2021) and four of the six matching-based tiers show no significant regression.
All learned variants in the revision keep the FiLM form, so the regression count is a statement about this adapter family (Section 6.3).
"It would help if the authors identified observations that were genuinely unexpected before running the benchmark, or cases in which existing QEC intuition would have predicted the wrong outcome."
Thank you for this suggestion. Three results run against the intuition that a structured fault changes the decision as soon as it reweights competing explanations.
-
A fault that mimics a likelier error shifts the rule later. The aligned pair crosses the 4% level at a higher strength than the horizontal pair (0.012 against 0.0075 at d = 3, r = 2), because its decision flips only once the pair is more likely than the single error it mimics. The submission had recorded the opposite prediction before the run (Appendix G).
-
More changed decisions do not mean more cost. At the benchmark strengths the grid changes the optimal decision on more probability mass (0.62 to 0.73%) than the horizontal pair does (0.37%), and it costs less than half as much, because the grid flips decisions that were near ties.
-
Rate changes cost little when all rates scale together, and more when only some do. Scaling every rate by the same factor costs at most 1.3% of target Bayes risk up to a multiplier of ten. Scaling only some axes crosses 4% at a multiplier of four (d = 3, r = 2, exact). The benchmark grid stays cheap because it stops at three.
"Main evidence comes from synthetic noise." / "Exact results are limited to relatively small QEC settings."
An exact optimal rule on both sides of a shift exists only for a specified noise model, so the hotspots and pairs are stress tests by design (Section 6.3). Section 5.2 and Appendix U compare the hotspot strengths with published single-qubit relaxation fluctuations, and the paper also analyses 420 recorded Willow configurations (Appendix S).
The central contrast is measured with exact references, not extrapolated. At d = 3, r = 8 and d = 5, r = 5 the largest evaluated grid point costs 4.4% and 5.8% of target Bayes risk, while the structural scenarios cost 19 to 173% (Table 1). At d = 7, r = 2 the revision resolves the aligned pair at 124% (95% interval 98 to 158%), against 10.4% for the largest grid point. The other structural cases at d = 7 remain unresolved.
We hope these responses address the reviewer's concerns, and we would be glad to clarify any remaining point during the discussion period.
Reference
- Wang, Shelhamer, Liu, Olshausen and Darrell. Tent: Fully test-time adaptation by entropy minimization. ICLR (2021).
Gentle Reminder
Dear Reviewer gEuC,
We sincerely appreciate your thoughtful review, particularly your suggestion on observability, which helped us strengthen the paper.
As the discussion period approaches its end, we would be grateful for your feedback on our response and revision. If any concerns remain, we would be happy to provide further clarification.
Thank you again for your time and consideration.
Response to Reviewer gEuC (1/2)
We sincerely thank the reviewer for the careful review and the insightful comments. We respond to each weakness and question, in two comments.
Thanks for your concern. The contribution is two-fold, and neither part is implied by the intuition or by existing theory. NSBench is a Datasets and Benchmarks track submission.
(1) A benchmark with exact references on both sides of a shift. In typical ML benchmarks test-time adaptation is scored without knowing the Bayes risk on either side of the shift. In QEC the circuit gives the exact joint distribution of syndrome and label, and in its tractable settings NSBench computes both optimal rules on the shots that every decoder sees, which turns the standard decomposition (Section 3.2) into a measurement. To the best of our knowledge, no existing benchmark for neural QEC decoders provides these two references. Its theory is the error identity of Appendix E and the identifiability rule of Appendix W.1.
(2) Findings that the intuition does not supply. The intuition says that both effects exist. It does not say which one the field's shifts produce, or how much of it unlabeled data can reveal.
The multiplier grid is mostly a difficulty shift. A grid up to three, as used in recent out-of-distribution tests, raises error rates several times over, yet keeping the source rule costs at most 3.8% of target Bayes risk in the dense exact settings. A decoder can look robust or adaptive on that grid without its decision rule being tested.
Part of the shift cost is invisible to unlabeled data. For the aligned pair, any rule that sees only syndromes loses at least 70% of the shift cost in one of two indistinguishable worlds (next answer).
Adaptation has a measurable price where little is recoverable. A learned adapter loses accuracy on small-cost scenarios, including the no-shift control, and the rise in error rate does not predict where (Appendix W.4).
We sincerely thank the reviewer for this valuable suggestion. The syndrome distribution fixes the firing rate of each detector footprint (the set of detectors a fault flips), but not how that rate splits between faults that flip the logical observable and faults that do not (Wagner et al., 2022, 2023, Blume-Kohout and Young, 2025, Zheng et al., 2026). Applied to decoding under shift, this gives three cases, stated as a rule in the revision (Section 5.4, Appendix W.1).
Recoverable in principle. A static shift that only reweights mechanisms of the source model.
Recoverable in our scenarios. A shift that adds a footprint whose syndrome the source rule already decodes with the correct logical value.
Hidden. A shift whose new fault produces a syndrome that the source rule decodes with the opposite logical value.
The bound holds for every label-free method without a prior separating the two worlds. A single-qubit X error at the same probability on a different qubit produces exactly the same syndrome distribution as the aligned pair, so any syndrome-only rule loses at least 70% of the aligned pair's shift cost in one of the two. A census confirms the rule without exception. Among all 129 X faults on one, two or three data qubits at d = 3, r = 2, no single-qubit fault is hidden, and 18 of the 36 pairs and 48 of the 84 triples are. The ambiguity is not special to d = 3. An exact census at d = 5, r = 1 finds no hidden fault among the 325 X faults of weight one or two, and 292 of the 2,300 of weight three, exactly the set that the standard correctability condition predicts. One has the same syndrome as a weight-two fault with the opposite logical value, with equal label-free estimates (2.13e-3) and exact costs of 2.13e-3 and 19.65e-3.
References
Wagner, Kampermann, Bruß and Kliesch. Quantum 6, 809 (2022).
Wagner, Kampermann, Bruß and Kliesch. Physical Review Letters 130, 200601 (2023).
Blume-Kohout and Young. arXiv:2504.14643 (2025).
Zheng, Chu, Chen, Manes, Lee, Zhou and Jiang. arXiv:2601.22286 (2026).