- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a benchmark for signal-like versus background-like event classification across four neutrinoless double-beta-decay detector datasets spanning waveforms, point clouds, and sparse images. It proposes an energy-weighted AUC and a histogram-based energy-independence score, then compares four native-representation models with nine variants of a shared Transformer backbone. The principal empirical claim is that Region + MLP is the most consistently strong configuration across modalities, while the broader resource contribution is a coordinated release and common evaluation framework.
Critical strengths
- The coordinated release of datasets from distinct detector technologies is potentially valuable to both scientific ML and experimental-physics communities.
- The paper is unusually transparent about what its two energy-aware metrics do and do not establish. I appreciate the honesty.
Critical weaknesses
- Every model is trained with one seed, many reported differences are only 0.001–0.01 AUC, and NEXT permits portions of the same source file in different splits. It would be helpful to include statistical analysis, e.g., multiple seeds.
- If could be wrong, but counting “Top-3” finishes across four heterogeneous proxies seems to be an arbitrary rank statistic, not evidence of a detector-general representation.
Questions for authors
- What experiment directly demonstrates that the classifiers use event topology rather than another non-topological correlate, given that neither AUCw nor the independence score identifies the predictive feature?
- How do the rankings change when models are trained on energy-matched samples and evaluated within narrow energy strata, alongside the current pooled, post-hoc weighted AUC?
- Are the 0.001–0.01 AUCw differences in Table 1 stable over independent training seeds, and what are the resulting confidence intervals and Top-3 membership frequencies?
- Why is “Top-3 count” an appropriate aggregate statistic when it discards effect size, uncertainty, and the very different difficulty and sample size of each dataset?
- Does Region+MLP’s Top-3 pattern persist when native architectures and Transformer variants are chosen with comparable, documented validation-only selection procedures and search budgets?
3 comments
I thank the authors for their thoughtful responses, which have addressed many of my initial questions. I can increase my score but I do believe the contributions of this paper are a better fit for a domain-specific journal than an AI conference.
We thank the reviewer for engaging with our responses, and for considering a higher score. We appreciated the flaws pointed out in the paper, and we spent considerable amount of time running experiments, learning from them, and updating our paper.
We originally submitted to the ICLR datasets and benchmarks area because we think these questions go beyond physics: both metrics apply to any nuisance variable measured for every event, not just energy (e.g. mass sculpting in collider jet tagging). We hoped the benchmark would be useful to the broader ML community. That being said, we will definitely keep in mind a more domain-specific journal for future work.
Thanks again for your time and constructive questions and responses.
We thank the reviewer for the specific and actionable questions. Please also see our general response.
W1 / Q3. Single seed and small differences.
All 13 models are now trained with 3 seeds on all 4 datasets (natives with 6). Seed-to-seed spread (SD) in AUCw is below 0.005 for most models. It is larger for Entity+MLP and the RoPE Summary/Entity variants on MJD (0.02–0.06). The test sets are large (117k–590k events), so bootstrap CIs over test events are much smaller than this. Seed spread is therefore the uncertainty that matters for rankings. In practice:
NEXT split. Our wording in Appendix A.3 overstated the issue. No event is ever divided between splits. Of 14,973 source files, none is shared between train and test. Two files are shared between validation and test, contributing 39 of 116,549 test events. Removing them changes inclusive AUC, AUCw and I by at most 0.0001 for every model and seed. We will correct the appendix.
W2 / Q4. Top-3 count.
We agree it discards effect size and uncertainty, and we have dropped it. Across seeds:
The full table covers all 13 models. We now describe Region+MLP as tied for the best mean and the most consistent across datasets, not as the strongest model.
Q1. Topology vs. non-topological correlate.
We scrambled the arrangement of each event and kept every sample value, and therefore its energy and amplitude distribution:
Energy alone gives the same AUCw before and after every scramble, as expected. On NEXT, I is also unchanged (0.969–0.974).
Q2. Narrow energy windows and energy-matched training.
(i) In a 100 keV window at Qββ, energy alone gives AUC = 0.50. The Region and Summary Transformers keep high AUC there: MJD 0.96–0.99, EXO-200 0.93–0.96, NEXT 0.98–0.99. The ranking inside the window is close to the overall one. On EXO-200 the three Region models are the top three, on MJD Region+RoPE and Region+Fourier lead, and on NEXT Region+MLP is the best Transformer.
(ii) We retrained Region+MLP and Summary+MLP on energy-matched training sets (one seed). On MJD the order is unchanged (0.965 vs 0.941). On EXO-200 Region+MLP's advantage disappears (0.953 vs 0.955, within seed noise).
Q5. Comparable selection procedures.
We re-selected the NEXT native candidates using validation data only. Validation and test rankings agree exactly within every architecture group (Spearman ρ = 1.0), so the test set did not change any choice. However, our stated rule ("best of each group") was not what we did: only GINE is its group's validation winner. The validation winners in the other groups (PointMLP 0.994, sparse 3D ResNet 0.991, CNN–GNN hybrid 0.991) all score above Region+MLP on NEXT (0.987). We will report them and correct the text. The native models also keep their own training settings (Table 9), so the comparison is not at an equal tuning budget. We therefore no longer claim that Region+MLP beats tuned modality-specific models on each dataset. Our claim is narrower: one shared backbone is competitive across four detector types.