- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors use a large dataset of simulated quantum circuits to compare models for noise prediction and error mitigation outside their training distribution. Trees using engineered features outperform the neural models on the widest circuits tested. On the held-out circuit family, the neural models do better, and selecting models by validation error gives a different ranking from test error which is interesting. Table 1 is very helpful and helped me understand how the benchmark compares with previous work, and the figures are easy to follow. My main reservation is that the widest circuits are still shallow, with targets that depend on local lightcones. Section 5.2 and Appendix B examine why the trees do well here, but I’m not sure how much to attribute to the models themselves rather than the features provided to them.
Critical strengths
The evaluation design is well done, especially the checks on whether low prediction error reflects useful learning when many targets are near zero.
S1. Using lightcone factorization to establish near-zero controls is a clever idea that I haven’t seen in learned QEM. The transformers predict larger correlations on these pairs than the measured inputs, despite the targets being close to zero. Looking only at aggregate MAE would make this difficult to see.
S2. The far buckets have explicit requirements for nonzero signal and improvement over predicting zero. This is helpful given the target distribution, and the authors also check for training runs that collapse to nearly constant predictions.
Critical weaknesses
W1. The family hold-out still includes circuits from the same generator in training. Appendix B explains that shallow Trotter circuits remain in the wide-shallow family, so I’m not convinced this tests transfer to a new circuit family as claimed. Splitting the test results by depth does not resolve that overlap. Since this experiment supports both the neural models’ advantage and the finding about validation rankings, I would want to see it repeated with all circuits from that generator excluded from training.
W2. The label audit found errors affecting 26% of the original far-width bucket. The authors exclude affected circuits from testing but retain affected training rows, with corrected labels planned for release. It is reassuring that the reported rankings survive the exclusions, but removing these cases changes the evaluation set and does not show what happens after correcting the data and retraining. For a paper whose main contribution is the benchmark, I would want the reference results to use the corrected version being released
Questions for authors
Q1 (Clarification). Table 9 partly addresses this by separating the held-out family into shallow and deeper circuits, but does the neural models' advantage persist when all Trotter circuits are removed from training, including those in the wide-shallow family?
Q2 (Clarification). How do the far-width results change when the affected labels are corrected and the excluded cases are restored, rather than removed from evaluation?
Q3 (Clarification). The paper examines the effect of removing width-dependent features, but how much of the trees' far-width advantage depends specifically on the engineered lightcone features?
Q4 (Suggestion). For the shallow far-width circuits, a comparison with directly simulating only the observable's lightcone, in terms of runtime and accuracy, would help clarify the practical advantage of learned prediction.
3 comments
Follow-up: NCGT and validation ranking without Trotter circuits in training
As promised in our reply to Q1 / W1, we add the NCGT and the validation–test rank correlation on the split with every Trotter circuit removed from training. Task B test MAE (×10⁻²):
| Model | Family hold-out (paper) | No Trotter circuits in training |
|---|---|---|
| Ridge | 6.41 | 6.30 |
| Random forest | 5.70 | 5.70 ± 0.01 |
| XGBoost | 5.52 | 5.61 |
| MLP (features) | 4.32 | 4.65 ± 0.21 |
| GatedGCN | 4.58 | 5.23 ± 1.03 |
| GraphGPS | 4.73 | 4.36 ± 0.52 |
| NCGT | 3.92 | 4.62 ± 0.34 |
The column "Family hold-out (paper)" repeats Table 2. In the new column, values with ± are the mean ± s.d. over three seeds. Ridge and XGBoost are deterministic, so they have one run.
The NCGT stays ahead of both tree ensembles on this stricter split (4.62 against 5.61 and 5.70).
Validation error, measured on circuits from the training families, ranks these seven models with Spearman ρ = 0.39 against test error. On the i.i.d. split, the same seven models give ρ = 0.96. The gap between validation and test rankings therefore remains when no Trotter circuit is seen in training.
Both results are in Appendix F.4 of the revised PDF, updated today.
Thank you for the additional experiments addressing the family overlap, lightcone features, and corrected labels. These resolve most of my concerns, and I will maintain my acceptance recommendation and score of 3. I would still encourage retraining the graph models on the corrected labels and adding a matched runtime comparison with direct lightcone simulation to clarify the practical benefit of learned prediction.
Response to Reviewer TVm2
We thank the reviewer for the detailed and constructive review, and for the comments on the lightcone-certified controls and the non-triviality criteria. We ran each experiment the reviewer asked for. Numbers are Task B test MAE (×10⁻², mean ± s.d. over seeds).
Main reservation and Q3. Do the trees' far-width results come from the models or from the engineered lightcone features?
From the models. We retrained the tree ensembles without the 12 lightcone features. These features are gate and qubit counts inside the depth-3 reverse lightcone of the measured qubits:
Without the lightcone features, the random forest is unchanged. Both trees stay ahead of every graph model, even when we also remove the width-dependent inputs. The far-width result is therefore a property of the learner: trees saturate inputs outside their training range (Section 5.2). It does not come from a locality prior in the features.
Q1 / W1. Family hold-out with every circuit from the Trotter generator removed from training.
We built this split as suggested. The test set is unchanged (all Trotterized-Ising circuits). We removed all 4,534 shallow Trotter circuits of the wide-shallow family from training and validation, so the Trotter generator is entirely unseen in training.
The neural models' advantage persists on this stricter split. The feature MLP, GatedGCN, and GraphGPS all stay ahead of both tree ensembles. The trees remain at the level of the constant-zero predictor (5.57). GraphGPS is even more accurate here than on the paper's split. The NCGT runs on this split are in progress. We will add them to this thread, together with the validation–test rank correlation.
Q2 / W2. Corrected labels with the excluded cases restored.
We re-evaluated the paper's trained models on the full far bucket. Every audited instance is restored with its corrected label and input (13,071 instances):
The ordering is identical. We also retrained every feature-based model on the corrected labels. On the far bucket the picture is the same (RF 1.80, XGBoost 1.77). On all five splits the conclusions of Table 2 hold:
The corrected labels are now the benchmark's reference labels (version v1.0.1). The revised PDF will include these tables.
Q4. Comparison with directly simulating the observable's lightcone.
We thank the reviewer for this suggestion and ran the comparison on 300 random far-bucket instances. Simulating only the observable's reverse lightcone reproduces the reference labels (largest deviation 7×10⁻⁵). Its cost depends on the lightcone width, which reaches 40 qubits in this sample. On one core, the median runtime is 0.03 s and the slowest instance takes 6 min. The cost grows exponentially with lightcone width. We placed the far bucket in this verifiable regime on purpose. Here QuOOD can score extrapolation to 2–3× the training width against exact labels. Learned surrogates are meant for circuits whose lightcones are too wide to simulate. Before relying on a model there, a practitioner can check its behavior in the verifiable regime and on the control stratum.