- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper presents the DecayFM model for classifying J/psi decay modes in high energy physics. DecayFM embeds kinematic, energy, and particle features separately, combines them with a transformer block, and finally uses a router + classification head. Results on classification, transfer learning, and few-shot learning are presented on a new rare decays dataset.
Critical strengths
- Interesting application to rare decays in high energy physics.
- Introduces a new, decently-sized dataset (which I hope the authors will release publicly).
- The paper is mostly well-written and the method and design choices are easily understood.
Critical weaknesses
Unfortunately, I have significant concerns regarding the experimental rigour, novelty, and significance of this work.
Soundness
- Overall the methodology is weakly motivated and not supported by empirical results and ablation studies. The authors discuss their reasoning for choices such as separate embeddings for subsets of features and the long-tail-aware router, but these are all based on assumptions that are never tested. It is important to test e.g. replacing the separate embeddings by a joint embedding, or removing the long-tail-aware router, etc. Some of the decisions such as separate embeddings may sound nice on paper but could in fact unnecessarily restrict the expressivity of the network while not reducing parameters.
- Many of the previous works mentioned such as MPM and JetCLR are self-supervised learning methods, so it is not clear why they would be good benchmarks for a supervised learning method such as DecayFM.
- The paper focuses entirely on supervised training on simulations with no discussion of the downstream use cases and how such a model would be calibrated for application to real data.
- Missing a lot of training details such as model sizes (both for DecayFM and benchmarks), training time, LoRA ranks (which should be ablated as well), etc.
- No error bars over multiple seeds.
Informativeness
- The model architecture is a basic combination of existing pieces and not novel. This could perhaps be compensated for by rigorous ablation studies, but those are not provided either. Thus, there are no clear conceptual or empirical insights gained from this work.
- It is expected that a pretrained model exposed to more training should outperform a model trained from scratch on a much smaller dataset. This has been demonstrated in myriad domains, included particle physics as well, so I’m not sure what the novelty is in the experimental results.
- The long-tail-aware router could be an interesting component, but it needs both ablation studies as well as engagement with prior work on routers, such as for MoE models, and empirical analysis of whether the router is actually working as expected.
- Overall, as the subfield of classification algorithms in particle physics is fairly mature now, it would be far more informative to start with an existing high-performing baseline such as ParT and test your new additions as modifications to the baseline one-by-one. By implementing many different things at the same time from scratch in DecayFM, it is hard to disentangle which aspects are responsible for any gains over existing baselines.
Presentation
- Overall, it is missing a high-level motivation for why we should be interested in rare J/psi decays specifically in the first place. I am not aware of previous publications in this particular subfield of particle physics at ICLR or similar venues, which is why a clear motivation is all the more important.
Questions for authors
I list my points of confusion and weaknesses for the authors to address above. Some additional questions:
- Can you support each of your design decisions with careful ablation studies?
- A lot of the design decisions are applicable to other particle physics datasets as well such as boosted jets at the LHC; can you thus also test your model on established datasets such as JetClass to more clearly benchmark it against existing baselines?
- Why focus specifically on J/psi decays? It seems an overstatement to term this a “foundation model” without demonstrating generalization to other particles.
- Will you release the dataset publicly?