- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The Limit Order Book (LOB) is the representation of financial market at its finest granularity. The paper concerns the task of training state-space models (Mamba-3) to autoregressively predict order flow given tokenized history data, and find out how the performance of such models scales with the total parameter count of the model, number of tokens used to train the model, and the total training compute (which can be derived from the former two). The authors trained and evaluated the models under various configurations, fitting additive models to find the scaling law equation, similar to the process of finding the scaling law for language modeling.
Critical strengths
The experiments in this work are done extensively and thoroughly, with the experiment settings are provided in detail, bolstering the reproducibility. The predicted scaling law can be useful as a guide for balancing model size with compute for training future larger LOB models.
Critical weaknesses
The optimization target is to minimize the cross entropy loss like autoregressive language models, given that the events are all tokenized. However, since a large part of financial data is numeric, cross entropy does not seem to be a good proxy metric when evaluating the models. Under the current regime, generating 10,00 for an event's size fields when the groundtruth is 9,99 is punished more than when the model predicts 99,99 for those fields. Consequently, I am not sure whether the trends of the cross entropy loss is a good enough proxy for measuring how well the model models the market.
Questions for authors
- The main concern of mine is about the discrepancy between the target metric (cross entropy) and the actual use case of such models. Please refer to the "Weakness" section of my review.
- Apart from the total compute used for training the model, how does dataset selection affect the predictive performance? Specifically, since market data could contain regime shifts, will the introduction of more history data negatively affect the model because , or will it improve the model's robustness due to the model having seen more diverse patterns?
We thank the reviewer for raising this point, but we disagree that cross-entropy (CE) is inherently unsuitable for modelling numerical distributions. Our objective is to build a market simulator: the model outputs the full conditional distribution of the next event, from which we then sample. We therefore, don’t attempt to minimise the numerical distance between individual predicted and observed events, but to learn the underlying conditional probability distribution of order flow. CE is a strictly proper scoring rule, and its expectation is minimised when the predicted distribution matches the true distribution, regardless of whether the outcomes represent categorical or numerical quantities. However, the reviewer’s example of predicting 1,000 instead of 999 being penalised more than predicting 9999 is a reasonable concern not for the objective itself but actually for the tokenisation. More precisely, CE penalises the probability assigned to the ground-truth tokens, rather than numerical distance, and our tokenisation does not explicitly preserve numerical ‘proximity’. For instance, moving from 999 to 1000 changes both size tokens, whereas moving from 999 to 9999 changes only one. We agree that this could affect learning efficiency, but it does not change the distributional consistency of the objective.
\text{Size} & \text{Count} & \text{Share} \\\\ \hline 99 & 24{,}304 & 0.021\\% \\\\ \mathbf{100} & \mathbf{80{,}837{,}341} & \mathbf{68.6\\%} \\\\ 101 & 3{,}917 & 0.003\\% \\\\ 999 & \mathbf{28} & 0.00002\\% \\\\ 1{,}000 & \mathbf{14{,}993} & 0.013\\% \\\\ 1{,}001 & 16 & 0.00001\\% \\\\ 9{,}999 & 612 & 0.0005\\% \end{array}$$ This table is measured by distribution of order sizes in GOOG, January 2025, 20 trading days, 117.9M messages. Multiples of 100 make up 72% of all non-zero sizes. Size 999 occurs only 28 times, while size 1,000 occurs 14,993 times, about 535 times as often. Two numerically adjacent values can differ in true probability by orders of magnitude. The assumption that "999 and 1,000 are numerically close, so they should receive similar penalties" does not hold. In real LOB data, sizes concentrate on round lots, and the distribution is far from smooth. The true distribution itself assigns very different probabilities to 999 and 1,000. A loss that gives "partial credit" by numerical distance would therefore push the model towards the wrong distribution. Only a proper scoring rule that scores probabilities, such as CE, can learn this spiky structure. We also agree that CE alone isn’t sufficient to establish realism of generated market trajectories, which is why we also evaluate closed-loop generation using LOB-Bench (https://arxiv.org/abs/2502.09172v2), measuring distributional realism across 21 market features using Wasserstein distance (Fig 3), alongside KS and L1 (appendix E). We therefore view CE as an appropriate metric for studying the scaling of conditional order-flow prediction, whilst treating closed-loop distributional realism as a separate and complimentary measure of generative quality. The question regarding the dataset distribution is interesting and important, but is slightly outside the scope of the paper, as the composition and temporal distribution of the training data isn’t captured by the exposure parameter D in our scaling predictions. Our experiments do not isolate the effect of adding older historical data whilst holding other variables fixed, so we can’t draw a rigorous conclusion on regime shifts. That said, for the reviewer's interest, in our other experiments we have generally observed that increasing the amount of historical market data improves downstream performance. Interestingly (and perhaps somewhat intuitively from a finite capacity perspective), we have also found that expanding the dataset by including more stocks can sometimes decrease performance on individual stocks. These are, of course, distinct ways of increasing the training data, and we haven't presented controlled evidence for these observations here, so they should be taken with a grain of salt. We mention them simply out of scientific curiosity, as we agree that the question of data composition is an interesting direction for further research.