- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
The paper applies residual vector quantization and autoregressive transformers to simulate collider event data. The authors argue that token prediction error improvements translate into improvements in event level and jet level residuals for accurate physics simulations. The authors also conduct experiments to suggest that token level scaling laws are a proxy for estimating physical performance.
Critical strengths
I think the idea of applying tricks from LLM architecture (RVQ, Autoregressive Transformers) to collider event generation is an interesting one. There is a need to simulate realistic collider event data and finding ways to leverage generative modeling is an interesting angle.
Critical weaknesses
This is an application paper, RVQ and autoregressive transformer models are standard and so there isn't any ML novelty. The connection between two pretty niche areas (LLM architecture and experimental particle physics) is unclear. Since this is the main contribution, I think the paper should spell out how applying these ideas lets practitioners do something new or useful. The writing is difficult to follow, and I'm not sure who the intended audience is for the paper. Even though it’s an application paper, the ideas from both ML and collider simulations should be self contained. I would suggest reviewing the existing generative models for collider event generation or analysis and adding a figure detailing the quantizer network.
Questions for authors
What exactly gets mapped to a fixed vocabulary, and how is that vocabulary defined? Section 2.1 says something about particles being defined by transverse momentum pT, pseudo-rapidity \eta, azimuthal angle \phi, its class and coordinates of its production vertex. These should all be defined for an ML audience. What is important about both top quark and anti-top quark pair production data vs QCD data? Are the inputs the same for all datasets? The paper says that the fidelity of the representation is limited by the codebook size K, how is K chosen and how is the number of residual quantization steps N optimized? How do tradeoffs between K and N impact downstream performance? There also seems to be no architectural novelty in the generative model itself, so comparing against other 3D point cloud generators is a class of important missing baselines.
3 comments
Thanks for the clarifications and ablations. My main concern is that as an application paper, the value of the paper depends on demonstrating an application. The paper shows that the surrogate reproduces distributions, but not that it lets physicists do something new or useful. Does it produce the same physics conclusions as full simulation in some downstream analysis (for example a measurement or search sensitivity with uncertainties)? Without this, the paper is of limited interest from an ML perspective, and it's not clear to me what it enables for physicists.
The added timing uses non-matched published numbers on older hardware. The authors acknowledge this as an upper estimate, so it doesn't establish a strong cost advantage. The new comparison shows Parnassus a continuous generator outperforming the model on particle and jet level responses, so the benefit of discrete tokenization also isn't demonstrated.
We thank the reviewer for the continued engagement. We want to clarify that we do not claim the model is an analysis-ready replacement for the CMS full simulation today, but rather a proof of concept for residual-vector-quantized tokenization combined with autoregressive generation for fast full-event simulation. It shows that the approach works on CMS full-simulation data, and most importantly that it scales predictably and that improvements in token-level loss carry over to physically relevant metrics. To address the reviewer's question within the time constraints, we performed a limited, preliminary test on already-generated data of whether the model leads to the same analysis outcome as full simulation.
We use the same public CMS 2011 QCD and events released with Parnassus, and compare the full simulation, Delphes, Parnassus and our model, starting from the same generator-level event.
1 - Signal acceptance
Different analyses use different selection criteria for analyzed events, we select a variety of published CMS papers and take the selection cuts from them. For different criteria we compare the acceptance with the one derived from full simulation. We report the ratio to the full simulation .
\text{Selection} & \text{Sample} & \text{Delphes} & \text{Parnassus} & \text{AR 41M} & \text{AR 1B} \\\\ \hline \text{Dijet, } m_{jj}>0.9\text{ TeV [1]} & \text{QCD} & 1.103 \pm 0.003 & 0.993 \pm 0.003 & 0.978 \pm 0.003 & \mathbf{0.999 \pm 0.003} \\\\ H_T>1\text{ TeV [2]} & \text{QCD} & 1.275 \pm 0.004 & 0.988 \pm 0.004 & 0.984 \pm 0.004 & \mathbf{0.997 \pm 0.003} \\\\ \text{1 lepton} + \geq 4\text{ jets [3]} & t\bar{t} & 0.936 \pm 0.012 & 0.993 \pm 0.013 & 0.923 \pm 0.014 & \mathbf{1.001 \pm 0.012} \\\\ \geq 6\text{ jets [4]} & t\bar{t} & 0.929 \pm 0.027 & \mathbf{1.013 \pm 0.028} & 0.863 \pm 0.028 & 0.977 \pm 0.027 \\\\ \end{array}$$ Uncertainties are statistical, estimated by 200 bootstrap resamplings of the events. 2 - Impact on measurement Top quarks are produced mainly in $t\bar{t}$ pairs and decay almost always to a $W$ boson and a $b$ quark; when one $W$ decays to two quarks and the other to an electron or muon and a neutrino, the event has one lepton and four jets [3]. CMS uses these events to measure the $t\bar{t}$ production cross section (how often top-quark pairs are produced) as $\sigma = (N - B) / (\varepsilon \cdot L)$ [5], with $N$ the number of selected events, $B$ the expected background and $L$ the integrated luminosity. The acceptance $\varepsilon$ of this selection is taken from simulation. $N$, $B$ (in this setting) and $L$ do not depend on the simulator, so replacing full simulation by a model changes the measured cross section exactly by $\sigma_{\text{model}} / \sigma_{\text{full}} = \varepsilon_{\text{full}} / \varepsilon_{\text{model}}$: $$\begin{array}{l|c} \text{Acceptance from} & \sigma_{\text{model}} / \sigma_{\text{full}} \\\\ \hline \text{AR 1B} & \mathbf{0.999 \pm 0.012} \\\\ \text{Parnassus} & 1.007 \pm 0.013 \\\\ \text{Delphes} & 1.068 \pm 0.014 \\\\ \text{AR 41M} & 1.083 \pm 0.016 \\\\ \end{array}$$ With our 1B model the measured cross section is unchanged within our statistical precision, a shift of about 1% of the systematic uncertainty CMS quotes for this measurement (+8.2 / −9.2%, [3]). We want to stress that this analysis is done without energy-corrected jets, and leave calibration studies for further work. [1] arXiv:1210.2387 [2] arXiv:1208.2931 [3] arXiv:1212.6682 [4] arXiv:1302.0508 [5] arXiv:1208.2671
We thank the reviewer for the constructive criticism and the useful comments, in particular making it clear that some tools, features and metrics were not properly explained in the main text. We addressed this by explaining more in depth the motivation behind this work and we tried to explain more clearly what every feature or metric we referred to means.
Following, we address the main concerns and questions:
Methodology clarification
Additional ablations & comparisons
In a vanilla VQ-VAE the codebook size represents the number of possible configurations that the tokens can describe. As we add residual layers, each layer multiplies the possible number of configurations by its own codebook size. Thus the number of configurations that can be described by the tokenized representation balloons quickly when adding residual layers. The optimal codebook size and number of residual steps should be chosen by taking the following factors into consideration:
We added quantizer ablations to the appendix (A.2.1, Table 7, Figure 9), showing that, for our dataset, the quantizer we selected performs similarly or better than others with different K and M parameters trained with the same setup. We report here a table of the SWDs between the original and reconstructed events for the ablations.
M\times K & \text{Bits} & \text{Event} & \text{Jet} & \text{Particle} \\\\ \hline 3\times 8192 & 39 & 0.0158 & 0.0819 & 0.0476 \\\\ 4\times 4096 & 48 & 0.0111 & 0.0511 & 0.0348 \\\\ 5\times 776 & 48 & 0.0299 & 0.2955 & 0.0751 \\\\ 6\times 256 & 48 & 0.0936 & 0.2977 & 0.1042 \\\\ 4\times 8192\\ \text{(main)} & 52 & \mathbf{0.0073} & \mathbf{0.0382} & \mathbf{0.0275} \\\\ 5\times 1351 & 52 & 0.0351 & 0.0931 & 0.0476 \\\\ 6\times 406 & 52 & 0.0728 & 0.1059 & 0.0509 \\\\ 5\times 8192 & 65 & 0.0171 & 0.0637 & 0.0457 \\\\ 6\times 8192 & 78 & 0.1292 & 0.1017 & 0.0536 \end{array}$$We added to the appendix a comparison between our 41M autoregressive model and a discrete DLM-style model using the same backbone with bidirectional attention, trained for the same number of steps and with the same exact training recipe and both using the same quantizer. We find that the autoregressive generator performs better even at a high number of steps for the DLM-style model.
We added comparisons to both Delphes, a widely used parametric simulation, and to Dreyer et al., 2026. Our model is competitive with both on most metrics, and while Parnassus is more performant in modeling the conditional response to the inputs at the particle and jet-levels, our model excels in modelling the geometric properties of the detector’s response and artifacts induced by the Particle Flow reconstruction algorithm that both use as a target, as well as improving on event-level features. Given better reconstruction from the tokenizer, which seems to be the main limiting factor in modeling quality, we expect our generator to at least reduce the gap in the features where it is being held back. Please refer to Section 3.2 and Figures 4 and 5.