acceptodds
Under review as a conference paper at ICLR 2027

QuOOD: Benchmarking Out-of-Distribution Property Prediction on Quantum Circuits

Abstract

Machine-learned surrogates that predict the noisy outputs of quantum circuits or mitigate their errors learn from classical simulation, whose cost grows exponentially with the number of qubits for general circuits. They are trained on circuits small or structured enough to simulate and deployed on circuits that are wider, deeper, from new families, or run on new devices. Distribution shift is therefore the normal deployment condition. Existing datasets test these surrogates in distribution or one shift at a time. We introduce QuOOD, a benchmark that makes performance under these shifts measurable: 99,386 circuits and 394,684 instances, split into an i.i.d. control and four shifts in depth, width, circuit family, and device noise, with every test regime kept within the range where simulation still gives exact or error-bounded labels. Because many target values are close to zero, a low average error can hide both a model that predicts nothing and one that invents structure. QuOOD checks its farthest test regimes for signal and for a clear margin over predicting zero, and adds a control stratum of qubit pairs whose correlations the circuit's structure certifies as near zero. Across thirteen predictors, tree ensembles and neural models swap places: trees over engineered features lead every neural model at 2–3× the training width, where each target depends only on a bounded set of gates, and trail them on the held-out family. Graph networks fall further behind as width grows, even our noise-conditioned transformer with its width inputs removed. On the controls, the transformers (ours and GraphGPS) emit more spurious correlation than their measured input carries, whereas the trees shrink it. Our transformer leads on held-out calibrated devices through per-block noise conditioning. On the held-out family, validation error stops tracking the test ranking (Spearman ρ = 0.00, against 0.99 in distribution), so models meant for shifted deployment must be tested under shift, which QuOOD enables. We plan to release the dataset, splits, labels, and reference runs upon acceptance.

Primary area:
datasets and benchmarks
Keywords:
Out-of-Distribution Generalization, Quantum Machine Learning, Quantum Circuits, Quantum Error Mitigation, Datasets and Benchmarks

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_TVm2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors use a large dataset of simulated quantum circuits to compare models for noise prediction and error mitigation outside their training distribution. Trees using engineered features outperform the neural models on the widest circuits tested. On the held-out circuit family, the neural models do better, and selecting models by validation error gives a different ranking from test error which is interesting. Table 1 is very helpful and helped me understand how the benchmark compares with previous work, and the figures are easy to follow. My main reservation is that the widest circuits are still shallow, with targets that depend on local lightcones. Section 5.2 and Appendix B examine why the trees do well here, but I’m not sure how much to attribute to the models themselves rather than the features provided to them.

Critical strengths

The evaluation design is well done, especially the checks on whether low prediction error reflects useful learning when many targets are near zero.

S1. Using lightcone factorization to establish near-zero controls is a clever idea that I haven’t seen in learned QEM. The transformers predict larger correlations on these pairs than the measured inputs, despite the targets being close to zero. Looking only at aggregate MAE would make this difficult to see.

S2. The far buckets have explicit requirements for nonzero signal and improvement over predicting zero. This is helpful given the target distribution, and the authors also check for training runs that collapse to nearly constant predictions.

Critical weaknesses

W1. The family hold-out still includes circuits from the same generator in training. Appendix B explains that shallow Trotter circuits remain in the wide-shallow family, so I’m not convinced this tests transfer to a new circuit family as claimed. Splitting the test results by depth does not resolve that overlap. Since this experiment supports both the neural models’ advantage and the finding about validation rankings, I would want to see it repeated with all circuits from that generator excluded from training.

W2. The label audit found errors affecting 26% of the original far-width bucket. The authors exclude affected circuits from testing but retain affected training rows, with corrected labels planned for release. It is reassuring that the reported rankings survive the exclusions, but removing these cases changes the evaluation set and does not show what happens after correcting the data and retraining. For a paper whose main contribution is the benchmark, I would want the reference results to use the corrected version being released

Questions for authors

Q1 (Clarification). Table 9 partly addresses this by separating the held-out family into shallow and deeper circuits, but does the neural models' advantage persist when all Trotter circuits are removed from training, including those in the wide-shallow family?

Q2 (Clarification). How do the far-width results change when the affected labels are corrected and the excluded cases are restored, rather than removed from evaluation?

Q3 (Clarification). The paper examines the effect of removing width-dependent features, but how much of the trees' far-width advantage depends specifically on the engineered lightcone features?

Q4 (Suggestion). For the shallow far-width circuits, a comparison with directly simulating only the observable's lightcone, in terms of runtime and accuracy, would help clarify the practical advantage of learned prediction.

3 comments
AuthorsOpenReview ↗

Response to Reviewer TVm2

We thank the reviewer for the detailed and constructive review, and for the comments on the lightcone-certified controls and the non-triviality criteria. We ran each experiment the reviewer asked for. Numbers are Task B test MAE (×10⁻², mean ± s.d. over seeds).

Main reservation and Q3. Do the trees' far-width results come from the models or from the engineered lightcone features?

From the models. We retrained the tree ensembles without the 12 lightcone features. These features are gate and qubit counts inside the depth-3 reverse lightcone of the measured qubits:

Far bucketAll 70 features− 12 lightcone features− lightcone features − 3 width inputs
Random forest1.72 ± 0.021.73 ± 0.001.75 ± 0.01
XGBoost1.721.802.08
Best graph model (NCGT + features)2.34

Without the lightcone features, the random forest is unchanged. Both trees stay ahead of every graph model, even when we also remove the width-dependent inputs. The far-width result is therefore a property of the learner: trees saturate inputs outside their training range (Section 5.2). It does not come from a locality prior in the features.

Q1 / W1. Family hold-out with every circuit from the Trotter generator removed from training.

We built this split as suggested. The test set is unchanged (all Trotterized-Ising circuits). We removed all 4,534 shallow Trotter circuits of the wide-shallow family from training and validation, so the Trotter generator is entirely unseen in training.

ModelFamily hold-out (paper)No Trotter circuits in training
Ridge6.416.30
Random forest5.705.70 ± 0.01
XGBoost5.525.61
MLP (features)4.324.65 ± 0.21
GatedGCN4.585.23 ± 1.03
GraphGPS4.734.36 ± 0.52

The neural models' advantage persists on this stricter split. The feature MLP, GatedGCN, and GraphGPS all stay ahead of both tree ensembles. The trees remain at the level of the constant-zero predictor (5.57). GraphGPS is even more accurate here than on the paper's split. The NCGT runs on this split are in progress. We will add them to this thread, together with the validation–test rank correlation.

Q2 / W2. Corrected labels with the excluded cases restored.

We re-evaluated the paper's trained models on the full far bucket. Every audited instance is restored with its corrected label and input (13,071 instances):

Far bucket (width split)Cleaned set (paper, 9,721)All instances, corrected labels (13,071)
Random forest1.721.78 ± 0.02
XGBoost1.721.81
NCGT + features2.342.38 ± 0.25
NCGT2.392.39 ± 0.15
ML-QEM GNN2.972.93 ± 0.42
GatedGCN3.843.75 ± 0.93
GraphGPS4.153.83 ± 0.83

The ordering is identical. We also retrained every feature-based model on the corrected labels. On the far bucket the picture is the same (RF 1.80, XGBoost 1.77). On all five splits the conclusions of Table 2 hold:

Retrained on corrected labelsi.i.d.DepthWidthFamilyDevice
Random forest1.912.021.595.702.38
XGBoost2.082.151.645.632.70
MLP (features)1.851.971.394.342.18

The corrected labels are now the benchmark's reference labels (version v1.0.1). The revised PDF will include these tables.

Q4. Comparison with directly simulating the observable's lightcone.

We thank the reviewer for this suggestion and ran the comparison on 300 random far-bucket instances. Simulating only the observable's reverse lightcone reproduces the reference labels (largest deviation 7×10⁻⁵). Its cost depends on the lightcone width, which reaches 40 qubits in this sample. On one core, the median runtime is 0.03 s and the slowest instance takes 6 min. The cost grows exponentially with lightcone width. We placed the far bucket in this verifiable regime on purpose. Here QuOOD can score extrapolation to 2–3× the training width against exact labels. Learned surrogates are meant for circuits whose lightcones are too wide to simulate. Before relying on a model there, a practitioner can check its behavior in the verifiable regime and on the control stratum.

AuthorsOpenReview ↗

Follow-up: NCGT and validation ranking without Trotter circuits in training

As promised in our reply to Q1 / W1, we add the NCGT and the validation–test rank correlation on the split with every Trotter circuit removed from training. Task B test MAE (×10⁻²):

ModelFamily hold-out (paper)No Trotter circuits in training
Ridge6.416.30
Random forest5.705.70 ± 0.01
XGBoost5.525.61
MLP (features)4.324.65 ± 0.21
GatedGCN4.585.23 ± 1.03
GraphGPS4.734.36 ± 0.52
NCGT3.924.62 ± 0.34

The column "Family hold-out (paper)" repeats Table 2. In the new column, values with ± are the mean ± s.d. over three seeds. Ridge and XGBoost are deterministic, so they have one run.

The NCGT stays ahead of both tree ensembles on this stricter split (4.62 against 5.61 and 5.70).

Validation error, measured on circuits from the training families, ranks these seven models with Spearman ρ = 0.39 against test error. On the i.i.d. split, the same seven models give ρ = 0.96. The gap between validation and test rankings therefore remains when no Trotter circuit is seen in training.

Both results are in Appendix F.4 of the revised PDF, updated today.

Thank you for the additional experiments addressing the family overlap, lightcone features, and corrected labels. These resolve most of my concerns, and I will maintain my acceptance recommendation and score of 3. I would still encourage retraining the graph models on the corrected labels and adding a matched runtime comparison with direct lightcone simulation to clarify the practical benefit of learned prediction.

Reviewer_wdK2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work establishes a benchmark for evaluating the performance of different quantum-property-prediction models on out-of-distribution (OOD) data, covering both noisy expectation prediction and learning-based error mitigation. The benchmark includes traditional machine learning models and neural-network-based models, and considers several aspects of OOD generalization, including circuit depth, width, circuit family, and device noise. The results show that the relative performance of tree-based and neural-network-based methods depends on the type of distribution shift.

Critical strengths
  • This work is valuable for systematically auditing the OOD generalization performance of existing methods, shedding light on the actual progress of quantum-property-prediction models.

  • The selection of tasks and models is sufficiently broad.

Critical weaknesses
  • The benchmark only considers single-qubit ZZ and two-qubit ZZZZ observables. Although these observables cover different qubit separations, a more diverse set of observables, such as higher-order Pauli strings, would provide a more comprehensive evaluation.

  • The width-OOD task evaluates whether models can generalize to circuits with more qubits, but the current evaluation focuses on single-qubit and two-qubit observables. It remains unclear whether the observed generalization extends to observables whose effective lightcones grow with system size.

Questions for authors
  • As noted in Line 744, some labels were incorrectly generated. Were samples with incorrect labels also excluded from the training sets?

  • The family-OOD evaluation holds out the Trotterized Ising family, while HEA is also a part of the dataset. Could the authors report the results on HEA separately?

  • What insights or design rules for future quantum-property-prediction models can be derived from this benchmark?

4 comments
AuthorsOpenReview ↗

Response to Reviewer wdK2

We thank the reviewer for the careful reading and for valuing a systematic OOD audit of quantum-property predictors. We answer each point with new results. Numbers are Task B test MAE (×10⁻², mean ± s.d. over seeds) unless noted.

Q1. Were samples with incorrect labels also excluded from the training sets?

Affected circuits are excluded from every test set. We have now also retrained every feature-based model on the corrected labels (v1.0.1), on all five splits. The conclusions of Table 2 are unchanged on every split:

SplitRFXGBoostMLPRidgePaper (RF / XGB / MLP)
i.i.d.1.912.081.85 ± 0.033.801.93 / 2.14 / 1.87
Depth2.022.151.97 ± 0.225.692.17 / 2.24 / 3.09 (median 1.88)
Width1.591.641.39 ± 0.013.281.58 / 1.66 / 1.42
Family5.705.634.34 ± 0.716.415.70 / 5.52 / 4.32
Device2.382.702.18 ± 0.214.212.40 / 2.74 / 2.22

We also restored all audited instances of the width split's far bucket under their corrected labels (13,071 instances). The tree ensembles reach 1.78–1.81, and the NCGT and its hybrid reach 2.38–2.39. The ordering is the same as in the paper.

Q2. Results on the hardware-efficient ansatz (HEA) family separately.

We held out the HEA family with the protocol of the paper's family hold-out. We train and validate on all other regression families:

Held-out HEAConstant zeroIdentity (measured value)RidgeRFXGBoostMLPNCGT
MAE9.923.723.964.905.224.75 ± 0.254.15 ± 0.29

The NCGT is ahead of both tree ensembles and the feature MLP on this family. GatedGCN and GraphGPS will follow in this thread.

We also applied the same protocol to every regression family, and the paper's family-shift finding generalizes. The neural predictor is at or ahead of both tree ensembles on five of the six held-out families (QAOA: MLP 2.04 vs XGBoost 2.40; GHZ/QFT/Clifford+T: 2.61 vs RF 2.68). The trees lead only on the wide-shallow family. This family extends to 40 qubits, so it is the far-width regime of Section 5.2.

Q3. Insights and design rules for future predictors.

  1. Select models under the shift they will face. In distribution, validation error ranks the models almost perfectly (Spearman ρ = 0.99). Under the family hold-out it carries no information (ρ = 0.00). An i.i.d. leaderboard therefore cannot pick the best surrogate for a new circuit class.
  2. Under width extrapolation, out-of-range inputs decide accuracy. Tree ensembles saturate inputs beyond their training range. ReLU networks extrapolate them linearly. Without the three width-dependent inputs, the MLP closes most of its far-width gap (Section 5.2).
  3. Condition every layer on the noise. Per-block noise conditioning (FiLM) gives the transformer its lead on unseen calibrated devices (p = 0.0001). Without it, the far-bucket error rises by 71%.
  4. Audit outputs on certified controls. The lightcone controls reveal spurious correlation that aggregate MAE hides.
  5. Stabilize dense-attention regressors. A final LayerNorm and a zero-initialized readout remove the constant-prediction collapse (0 of 133 runs, Section 5.6).

W1. Higher-order Pauli strings.

QuOOD uses one- and two-body Z observables for a reason. Lightcone factorization certifies near-zero controls on two-body correlators (Section 3.4), and these controls let the benchmark expose invented signal. Observables are stored as qubit tuples, and the simulators compute Z-string expectation values of any weight. The graph models read any observable as a mask over measurement nodes. Higher-weight observables therefore reuse the same generator, simulators, and models.

W2. Observables whose effective lightcones grow with system size.

The width split contains this regime. Its test set has two parts. The first part holds the 13–20-qubit circuits of the algorithmic families. Their reverse lightcones grow with the system: in 81% of these instances, the observable's lightcone covers every qubit of the circuit. The second part is the wide-shallow family at ≥ 25 qubits. Its lightcones are bounded by depth and cover a median of 59% of the qubits. The ranking differs between the two parts:

Width-split test region (train ≤ 12 qubits)InstancesBest treeBest neural model
Algorithmic families, 13–20 qubits (growing lightcones)139,259RF 1.64MLP / NCGT + features 1.38 (p = 0.02)
Wide-shallow, ≥ 25 qubits (bounded lightcones, far bucket)9,721RF / XGBoost 1.72NCGT + features 2.34 (p = 0.005)

Neural models lead where lightcones grow with the system. Trees lead where every target depends on a lightcone bounded by depth. QuOOD covers both regimes. In the far bucket, extrapolation to 2–3× the training width can be scored against exact labels.

AuthorsOpenReview ↗

Follow-up: all graph models on the held-out HEA family

As promised in our reply to Q2, we add GraphGPS and GatedGCN on the held-out hardware-efficient (HEA) family. Each graph model now has three seeds. Task B test MAE (×10⁻², mean ± s.d. over seeds):

Held-out HEAConstant zeroIdentity (measured value)RidgeRFXGBoostMLPNCGTGraphGPSGatedGCN
MAE9.923.723.964.90 ± 0.025.224.75 ± 0.254.12 ± 0.214.90 ± 0.509.67 ± 5.23

Ridge and XGBoost are deterministic, so their three seeds give the same value.

With its third seed, the NCGT moves from 4.15 ± 0.29 (two seeds, our first reply) to 4.12 ± 0.21. It stays ahead of both tree ensembles, the MLP, GraphGPS, and GatedGCN.

The three GatedGCN seeds reach 6.54, 15.71, and 6.77. The 15.71 seed has the lowest validation error of the three (validation MAE 2.59, against 2.77 and 2.91). Here, validation error on the training families does not predict accuracy on the new family. The paper finds the same for the ranking of models. Under the family hold-out, validation error ranks the models with Spearman ρ = 0.00 against test error, compared with 0.99 in distribution (Section 5.1).

These results are in Appendix F.5 of the revised PDF, updated today.

Q1. Were samples with incorrect labels also excluded from the training sets?

The original training sets keep the samples with incorrect labels (Appendix A). The table in our first reply shows the effect of retraining the feature-based models on the corrected labels. The conclusions of the paper's main table (Table 2) hold on all five splits (Appendix F.1).

We thank the authors for their efforts in addressing my concern.

Q1. The authors' responses fully answered my question.

Q2. The additional experimental results fully answered my question but raised a new one. The results show that all learned models perform worse than the identity baseline. Does this indicate that learned error mitigation fails to improve upon noisy measurements under this family shift? It would be insightful to discuss this observation.

W1. The authors' response explains the reason for considering only one- and two-body Z observables in the benchmark. However, since the main contribution of this work is to build a benchmark, I think the lack of more diverse types of observables limits the scope and value of the benchmark.

W2: The authors' response clarifies that the width split includes a regime where observables' lightcones grow with system size, which partially addresses my concern. However, the far-width evaluation is restricted to circuits with depth-bounded lightcones. Although this restriction may be due to the computational cost of simulating wider and deeper circuits, it still limits the conclusions about width generalization to larger-scale circuits with growing lightcones.

Overall, the rebuttal improves the quality of this work, but I do not think it sufficiently addresses my concerns to meet the acceptance bar.

AuthorsOpenReview ↗

Learned mitigation against the measured value under family shift

We thank the reviewer for the follow-up and are glad that Q1 and Q2 are answered.

New question on Q2: does learned mitigation fail to improve on the noisy measurement under this family shift?

On the held-out hardware-efficient family, no learned model beats the measured value (identity, 3.72). The closest are Ridge (3.96) and the NCGT (4.12 ± 0.21).

The outcome depends on which family is held out. We held out each regression family in turn (Table 18, Task B test MAE ×10⁻²). Table 18 covers the feature-based models. On four of the six families, the best of them cuts the error of the measured value by 21–34%:

Held-out familyIdentity (measured value)Best feature-based model
QAOA3.092.04 (MLP)
Trotterized Ising6.154.32 (MLP)
GHZ / QFT / Clifford+T3.972.61 (MLP)
Wide-shallow2.321.84 (random forest)
Hardware-efficient3.723.96 (Ridge)
Random layered4.134.34 (MLP)

On all five splits of the main table (Table 2), the best learned model cuts this error by 36–60%. In distribution it falls from 4.07 to 1.78. Under the paper's family hold-out it falls from 6.15 to 3.92 (NCGT).

So learned mitigation improves on the noisy measurement under most shifts. It does not do so on every new circuit family, and an in-distribution evaluation cannot tell these cases apart. In distribution, the models cut the error by more than half. Validation error on the training families also does not predict the ranking on a new family (Section 5.1). For this reason, QuOOD reports the identity baseline on every split.

This observation also adds a design rule to our answer to Q3. Before applying a learned mitigator to a new circuit family, compare it with the unmitigated value on that family.

We added this discussion to Appendix F.5 of the revised PDF.

W1 and W2. We thank the reviewer for acknowledging our clarifications.

On W2, the width split does test width generalization with growing lightcones. Models train on circuits with at most 12 qubits. They are tested on 139,259 instances of 13–20-qubit circuits from the algorithmic families (random layered, QAOA, hardware-efficient, Trotterized Ising, GHZ / QFT / Clifford+T). In a sample of 600 of these instances, the observable's lightcone covers every qubit of the circuit in 81%. Here neural models lead in test MAE (×10⁻²): the MLP and NCGT + features both reach 1.38, against 1.64 for the random forest (p = 0.02). The far bucket adds 25–40-qubit circuits whose lightcones are bounded by depth. There, tree ensembles lead (1.72 against 2.34 for NCGT + features, p = 0.005). QuOOD thus measures width generalization in both lightcone regimes (Appendix F.3).

The far bucket (W2) and the observables (W1) follow one design principle: every prediction is scored against exact or error-bounded labels. This lets QuOOD separate missing signal from invented signal. On the certified controls, the transformer models predict correlations that the circuit's structure certifies as near zero (Section 5.4). Only verified labels expose this failure. Lightcone factorization certifies these controls for two-body correlators (Section 3.4), so QuOOD uses one- and two-body Z observables. The generator, simulators, and models handle Z-strings of any weight.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response to Reviewer Hr4a

We thank the reviewer for the thoughtful review and for recognizing the benchmark design and the empirical findings. We answer each question with new results. Numbers are Task B test MAE (×10⁻², mean ± s.d. over seeds) unless noted.

Q1. Additional held-out families, and a depth split controlled for family composition and width.

Additional held-out families. We repeated the family hold-out for every regression family. We train and validate on all other families:

Held-out familyIdentityRFXGBoostMLP
Random layered4.134.444.774.34 ± 0.46
QAOA3.092.642.402.04 ± 0.02
Hardware-efficient3.724.905.224.75 ± 0.25
Trotterized Ising (paper)6.155.705.524.32
GHZ / QFT / Clifford+T3.972.686.132.61 ± 0.39
Wide-shallow2.321.841.933.83 ± 0.62

The paper's reversal between trees and neural models generalizes across families. The neural predictor is at or ahead of both tree ensembles on five of the six held-out families. The trees lead only on the wide-shallow family. This family extends to 40 qubits, so it is the far-width regime of Section 5.2. On the hardware-efficient hold-out, the graph NCGT (4.15 ± 0.29) is also ahead of both tree ensembles and the MLP.

Controlled depth split. We built a depth split in which only depth changes. It uses three families whose depth is generated independently of width: random layered, hardware-efficient, and Trotterized Ising (correlation of depth and width ≈ 0). We split each family at its own median depth. Training and test sides contain the same families with the same width distribution (mean 11.97 vs 12.03 qubits, identical quartiles). The test circuits are deeper: up to 64 layers, against at most 34 in training.

Controlled depth splitRandom layeredHardware-eff.TrotterAll (84,690)
Constant zero3.805.926.255.22
Ridge4.824.017.435.58
Random forest4.643.096.244.84
XGBoost5.323.045.704.90
MLP (features)3.522.662.923.09 ± 0.04

With family composition and width held fixed, the neural predictor leads in every family (p < 10⁻⁴ against the random forest overall). The random forest falls to the constant-zero level on the deep Trotter circuits.

Q2. Sensitivity to the training budgets and model–split coverage; Task A.

Three results show that the conclusions do not depend on the budgets. (i) Inside the training range, the models fit equally well. On the width split's validation set (≤ 12 qubits), the dense-attention models reach 2.72–2.80 and the random forest 2.78 (Appendix D). The far-width gap therefore comes from extrapolation, not from fitting capacity. (ii) More data does not change the far-width ordering. The NCGT improves 4.4× from 4% to 100% of the training data, yet XGBoost trained on 4% is still ahead (Figure 2b). (iii) Under one fixed budget per model class, the ranking reverses across shifts. Trees lead at far width. Neural models lead under the family hold-out, on the controlled depth split above, and on unseen calibrated devices. A budget effect cannot produce such a reversal.

Task A and graph models on the new splits. We are running the graph models (NCGT, GraphGPS, GatedGCN) on the new splits of this response, and the NCGT and GraphGPS on the Task A family hold-out. We will add the results and the validation–test rank correlations to this thread.

Q3. How well do the simulated device shifts reflect real hardware, including drift and unseen noise mechanisms?

The device split covers the two ways a deployed model meets new hardware noise. New calibrations: the held-out calibrated devices have calibration vectors the model never saw, with values inside the training range. This is calibration drift on a known device family. A new mechanism: the coherent over-rotation setting adds a noise process absent from training. The two cases behave differently. Per-block noise conditioning helps significantly on unseen calibrations (FiLM ablation: 1.59 vs 1.71, p = 0.0001). On the unseen mechanism, the feature MLP leads. This gives a concrete rule for deployment. A noise-conditioned model follows calibration changes within its training range, and a new noise mechanism should be included in training. Every model reads its noise vector from backend calibration properties, so the trained models apply directly to a live backend.

W1 / W2. The width split also contains 13–20-qubit circuits whose lightcones grow with the system. There the neural models lead (MLP and NCGT + features 1.38 vs RF 1.64, p = 0.02). QuOOD thus covers both bounded and growing lightcones. The controlled depth split and the additional family hold-outs above extend the depth and family results.

AuthorsOpenReview ↗

General Response: revised PDF

We thank all reviewers for their careful and constructive reviews. We have uploaded a revised PDF. New or changed text is in blue. The main text stays within the 10-page limit. The new experiments are in a new Appendix F:

  • F.1: Feature-based models retrained on the corrected labels, and the far bucket with every audited instance restored (Tables 13 and 14). Reviewers TVm2, wdK2
  • F.2: Tree ensembles trained without the lightcone features (Table 15). Reviewer TVm2
  • F.3: The width split by lightcone regime. Lightcones grow with the circuit at 13 to 20 qubits and are bounded by depth at 25 qubits and above (Table 16). Reviewers wdK2, Hr4a, TVm2
  • F.4: Family hold-out with every Trotter circuit removed from training (Table 17). Reviewers TVm2, Hr4a
  • F.5: Every regression family held out in turn, including the hardware-efficient (HEA) family (Table 18). Reviewers wdK2, Hr4a
  • F.6: Depth split with fixed families and widths (Table 19). Reviewer Hr4a
  • F.7: Direct simulation of the observable's lightcone. Reviewer TVm2

We added short pointers to Appendix F in Sections 3.3, 5.1, and 5.2, in the caption of Table 2, and in Appendices A and B. The new experiments address two points of the Limitations: the compound family shift in the depth split, and the single held-out family. We removed these two points. We also updated the guidance table (Appendix E) and one sentence of the Conclusion. We will post further results in the individual threads and add them to the PDF.

AuthorsOpenReview ↗

Follow-up: graph models on the new splits and on Task A, and validation–test rank correlations

As promised in our reply to Q2, we add the graph models on the new splits, the NCGT and GraphGPS on the Task A family hold-out, and the validation–test rank correlations. Numbers are test MAE (×10⁻²), on Task B unless marked Task A. Each graph model has three seeds.

Controlled depth split. This is the split from our reply to Q1: three families, each split at its own median depth, with the same width distribution on both sides.

Controlled depth splitRandom layeredHardware-eff.TrotterAll
Constant zero3.805.926.255.22
Ridge4.824.017.435.58
Random forest4.643.096.244.84 ± 0.01
XGBoost5.323.045.704.90
MLP (features)3.522.662.923.09 ± 0.04
GatedGCN5.233.194.104.32 ± 0.30
GraphGPS3.772.943.943.63 ± 0.06
NCGT3.582.623.013.14 ± 0.13

The family columns give the mean over seeds. The last column gives mean ± s.d. over seeds. Ridge and XGBoost are deterministic, so they have one run.

The NCGT and GraphGPS have lower error than both tree ensembles in every family. Overall, both are ahead of the random forest (p = 0.002 and p < 0.001). With family composition and width held fixed, a neural model leads in every family.

Hardware-efficient family held out. With three seeds, the NCGT reaches 4.12 ± 0.21 (4.15 ± 0.29 over two seeds in our first reply). It stays ahead of the random forest (4.90 ± 0.02), XGBoost (5.22), and the feature MLP (4.75 ± 0.25).

Validation–test rank correlation. On each split, we rank the same seven learned models (Ridge, random forest, XGBoost, MLP, GatedGCN, GraphGPS, NCGT) twice: by validation error, measured on circuits from the training families, and by test error. The Spearman ρ between the two rankings is 0.96 in distribution. It is 0.25 with the hardware-efficient family held out, and 0.39 with the Trotterized Ising family held out and every Trotter circuit removed from training, including the shallow ones in the wide-shallow family.

Under both family hold-outs, validation error tracks the test ranking poorly, as on the paper's family split (ρ = 0.00, Section 5.1). The finding therefore holds for a second held-out family and without any Trotter circuit in training.

Task A, family held out. Task A predicts the noisy expectation value without the measured input. We ran the NCGT and GraphGPS on its family hold-out (Trotterized Ising, the Family column of Table 3).

Task A, family held outTest MAE
Ridge7.24
Random forest6.12 ± 0.01
XGBoost6.04
MLP (features)6.05 ± 0.02
GraphGPS5.81 ± 0.23
NCGT6.05 ± 0.06

GraphGPS has the lowest mean error. The NCGT matches XGBoost and the MLP.

The Task B results are in Appendix F.4–F.6 and the Task A results in Table 3 and Section 5.3 of the revised PDF, updated today.

AuthorsOpenReview ↗

Summary of the revisions and new results

We thank the reviewers and the AC. The new experiments support the paper's main findings. On the new family hold-outs, neural models match or beat the tree ensembles except on the far-width family. Validation error again tracks the test ranking poorly (ρ = 0.25 and 0.39 on two new family hold-outs). On the corrected labels, the conclusions of Table 2 are unchanged.

This note summarizes the new results from all threads. The latest revised PDF holds all of them, in blue, mostly in Appendix F. Numbers are Task B test MAE (×10⁻²) unless noted.

Family hold-out beyond one family: the findings hold (Hr4a, TVm2).

  • We held out every regression family in turn (F.5). The feature MLP matches or beats both tree ensembles on five of the six families. The trees lead only on the wide-shallow family, which is the far-width regime.
  • We removed every Trotter circuit from training (F.4). The NCGT (4.62), GraphGPS (4.36), and the MLP (4.65) stay ahead of both tree ensembles (5.61, 5.70).
  • Validation error ranks the seven learned models with Spearman ρ = 0.25 (hardware-efficient held out) and 0.39 (no Trotter circuits in training), against 0.96 in distribution. The paper's model-selection finding holds on both splits.

Depth split with fixed families and widths: a neural model leads in every family (Hr4a). Only depth changes. Both sides hold the same families with the same width distribution (F.6). Overall, the MLP (3.09) and the NCGT (3.14) are ahead of the random forest (4.84).

Corrected labels: conclusions unchanged (wdK2, TVm2). We retrained every feature-based model on the corrected labels and restored every audited far-bucket instance (F.1). The conclusions of Table 2 hold on all five splits. The far-bucket ordering of the models is unchanged.

Trees' far-width lead: it comes from the learner, not from the lightcone features (TVm2). Without the 12 lightcone features, the random forest is unchanged (1.73 vs. 1.72). Both trees stay ahead of every graph model, also without the width-dependent inputs (F.2).

Lightcone regimes: both are tested, with different winners (wdK2, Hr4a, TVm2). The width split tests both regimes (F.3). At 13–20 qubits, where lightcones grow with the circuit, neural models lead (1.38 against 1.64, p = 0.02). In the far bucket at 25–40 qubits, where lightcones are bounded by depth, trees lead (1.72 against 2.34, p = 0.005). Direct simulation of the observable's lightcone reproduces the reference labels on 300 far-bucket instances (largest deviation 7×10⁻⁵, F.7).

Learned mitigation: it helps on most held-out families and on every split (wdK2, Hr4a). On the held-out hardware-efficient family, the NCGT (4.12) is ahead of the random forest (4.90), XGBoost (5.22), and the MLP (4.75). The unmitigated measured value itself (3.72) is lowest there. Learned mitigation cuts the error of the measured value on four of the six held-out families (21–34%) and on every split of Table 2 (36–60%). QuOOD reports this identity baseline on every split. A family where mitigation does not help is therefore visible (F.5).

Task A under family shift: a transformer has the lowest mean error (Hr4a). GraphGPS and the NCGT now also run on the Task A family hold-out (Table 3). GraphGPS has the lowest mean error (5.81 against 6.04 for XGBoost).

Budgets and device shift: the far-width gap comes from extrapolation, and noise conditioning helps on new calibrations (Hr4a). Inside the training range, the dense-attention models and the random forest fit equally well. Their validation errors on the width split are 2.72–2.80 and 2.78. The far-width gap therefore comes from extrapolation. Per-block noise conditioning helps on unseen device calibrations (1.59 against 1.71, p = 0.0001).

Design rules: six rules, each backed by a QuOOD result (wdK2). Our replies give these rules for future predictors. One example: before applying a learned mitigator to a new circuit family, compare it with the unmitigated value on that family.

Reviewer wdK2 confirmed that Q1 and Q2 are fully answered. We thank all reviewers again for their suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.