acceptodds
Withdrawn from ICLR 2027

Scaling Laws for Order Flow Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Limit order books capture financial markets at their most granular level, and autore- gressive generative models have emerged as a powerful approach to modeling their dynamics, yet their scaling behavior remains poorly understood. We study how the size of state-space models (Mamba-3) and cumulative training-token exposure affect cross-entropy (CE) on S&P 500 order flow, examining both logged training CE from 2022–2025 and forward-time test CE on January 2026 data. We analyse 32 training lineages across 12 model sizes, spanning over two orders of magnitude (2.63–293.28 million parameters), with 249 January checkpoints evaluated on the same 487 tickers. Under the realised training budgets, January endpoint CE improves sharply across the smallest models and then varies non-monotonically with size. An additive parameter–exposure law predicts held-out lineages and sizes within the observed range more accurately when restrictive exponent bounds are relaxed. Most whole-size prediction error comes from extrapolating to the smallest model, while coefficient estimates remain sensitive to size coverage and fitting choices. We also evaluate generated continuations with LOB-Bench, comparing observed and generated event and book-state distributions after processing gener- ated events through a matching engine. We relate the resulting discrepancy scores to CE at matching checkpoints. Using activity-derived hardware-cost estimates, we compare IsoFLOP quadratics with the additive surface at fixed compute budgets. The results show where the additive law describes order-flow prediction and where its fitted exponents depend on the experiment and fitting choices.

Primary area:
foundation or frontier models, including LLMs
Keywords:
neural scaling laws, order flow generation, financial foundation models, compute-optimal training, state space models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YxmfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The Limit Order Book (LOB) is the representation of financial market at its finest granularity. The paper concerns the task of training state-space models (Mamba-3) to autoregressively predict order flow given tokenized history data, and find out how the performance of such models scales with the total parameter count of the model, number of tokens used to train the model, and the total training compute (which can be derived from the former two). The authors trained and evaluated the models under various configurations, fitting additive models to find the scaling law equation, similar to the process of finding the scaling law for language modeling.

Critical strengths

The experiments in this work are done extensively and thoroughly, with the experiment settings are provided in detail, bolstering the reproducibility. The predicted scaling law can be useful as a guide for balancing model size with compute for training future larger LOB models.

Critical weaknesses

The optimization target is to minimize the cross entropy loss like autoregressive language models, given that the events are all tokenized. However, since a large part of financial data is numeric, cross entropy does not seem to be a good proxy metric when evaluating the models. Under the current regime, generating 10,00 for an event's size fields when the groundtruth is 9,99 is punished more than when the model predicts 99,99 for those fields. Consequently, I am not sure whether the trends of the cross entropy loss is a good enough proxy for measuring how well the model models the market.

Questions for authors
  • The main concern of mine is about the discrepancy between the target metric (cross entropy) and the actual use case of such models. Please refer to the "Weakness" section of my review.
  • Apart from the total compute used for training the model, how does dataset selection affect the predictive performance? Specifically, since market data could contain regime shifts, will the introduction of more history data negatively affect the model because , or will it improve the model's robustness due to the model having seen more diverse patterns?
1 comment

We thank the reviewer for raising this point, but we disagree that cross-entropy (CE) is inherently unsuitable for modelling numerical distributions. Our objective is to build a market simulator: the model outputs the full conditional distribution of the next event, from which we then sample. We therefore, don’t attempt to minimise the numerical distance between individual predicted and observed events, but to learn the underlying conditional probability distribution of order flow. CE is a strictly proper scoring rule, and its expectation is minimised when the predicted distribution matches the true distribution, regardless of whether the outcomes represent categorical or numerical quantities. However, the reviewer’s example of predicting 1,000 instead of 999 being penalised more than predicting 9999 is a reasonable concern not for the objective itself but actually for the tokenisation. More precisely, CE penalises the probability assigned to the ground-truth tokens, rather than numerical distance, and our tokenisation does not explicitly preserve numerical ‘proximity’. For instance, moving from 999 to 1000 changes both size tokens, whereas moving from 999 to 9999 changes only one. We agree that this could affect learning efficiency, but it does not change the distributional consistency of the objective.

\text{Size} & \text{Count} & \text{Share} \\\\ \hline 99 & 24{,}304 & 0.021\\% \\\\ \mathbf{100} & \mathbf{80{,}837{,}341} & \mathbf{68.6\\%} \\\\ 101 & 3{,}917 & 0.003\\% \\\\ 999 & \mathbf{28} & 0.00002\\% \\\\ 1{,}000 & \mathbf{14{,}993} & 0.013\\% \\\\ 1{,}001 & 16 & 0.00001\\% \\\\ 9{,}999 & 612 & 0.0005\\% \end{array}$$ This table is measured by distribution of order sizes in GOOG, January 2025, 20 trading days, 117.9M messages. Multiples of 100 make up 72% of all non-zero sizes. Size 999 occurs only 28 times, while size 1,000 occurs 14,993 times, about 535 times as often. Two numerically adjacent values can differ in true probability by orders of magnitude. The assumption that "999 and 1,000 are numerically close, so they should receive similar penalties" does not hold. In real LOB data, sizes concentrate on round lots, and the distribution is far from smooth. The true distribution itself assigns very different probabilities to 999 and 1,000. A loss that gives "partial credit" by numerical distance would therefore push the model towards the wrong distribution. Only a proper scoring rule that scores probabilities, such as CE, can learn this spiky structure. We also agree that CE alone isn’t sufficient to establish realism of generated market trajectories, which is why we also evaluate closed-loop generation using LOB-Bench (https://arxiv.org/abs/2502.09172v2), measuring distributional realism across 21 market features using Wasserstein distance (Fig 3), alongside KS and L1 (appendix E). We therefore view CE as an appropriate metric for studying the scaling of conditional order-flow prediction, whilst treating closed-loop distributional realism as a separate and complimentary measure of generative quality. The question regarding the dataset distribution is interesting and important, but is slightly outside the scope of the paper, as the composition and temporal distribution of the training data isn’t captured by the exposure parameter D in our scaling predictions. Our experiments do not isolate the effect of adding older historical data whilst holding other variables fixed, so we can’t draw a rigorous conclusion on regime shifts. That said, for the reviewer's interest, in our other experiments we have generally observed that increasing the amount of historical market data improves downstream performance. Interestingly (and perhaps somewhat intuitively from a finite capacity perspective), we have also found that expanding the dataset by including more stocks can sometimes decrease performance on individual stocks. These are, of course, distinct ways of increasing the training data, and we haven't presented controlled evidence for these observations here, so they should be taken with a grain of salt. We mention them simply out of scientific curiosity, as we agree that the question of data composition is an interesting direction for further research.
Reviewer_XRh7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper trains Mamba-3 models for financial order flow generation across a range of model sizes and training token budgets. The authors use an existing scaling law to examine how model size and training exposure affect cross-entropy loss. The results show that the scaling law predicts the loss reasonably well within the observed range.

Critical strengths
  1. The experimental study is nontrivial, covering a wide range of model sizes and training budgets. The results highlight the importance of considering both model size and training exposure, rather than simply increasing model size to improve performance.

  2. The authors fit a scaling law to financial order flow data and evaluate its predictive performance through held out experiments and sensitivity analyses, demonstrating its usefulness within the observed range.

Critical weaknesses

1.The parameter scaling estimates are highly sensitive to fitting choices, raising concerns about the reliability of extrapolation to larger models: I appreciate the authors’ transparent reporting of boundary hits and sensitivity analyses. However, these results suggest that the current experiments do not reliably identify the parameter scaling exponent α. With the original fitting bounds, α reaches the upper bound of 2 in every January fit and all bootstrap draws (Tables 1). The original fitted coefficients also predict little improvement from increasing model size beyond roughly 30M parameters at fixed training exposure. It remains unclear whether this reflects the actual scaling behavior or the chosen parameterization. In particular, Appendix D reports a width independent term of approximately 1.88M parameters, accounting for 71.6% of the smallest model.

Questions for authors
  1. Appendix D shows that approximately 1.88M parameters are independent of model width. Could the authors examine whether using the width-dependent parameter count instead of the total parameter count changes the fitted scaling exponents and held-out prediction performance? This could help clarify whether the observed parameter scaling behavior is influenced by the choice of parameterization.

  2. Were the training windows shuffled across the entire 2022–2025 period? If not, could the smallest (2.63M) and largest (293.28M) models, which were trained on only around 17.7B tokens, have been exposed mainly to earlier market data? Could this partly explain the U-shaped endpoint CE in Figure 2, since these models may have seen less recent data before the January 2026 test period?

1 comment

We thank the reviewer for their comments and constructive questions. We agree that the sensitivity of the fitted scaling exponents is an important concern (and indeed, one of the main points of the paper), but the reviewer also raises an interesting additional point regarding the ‘width-independent’ parameter contribution, which we have now investigated.

Q1: As noted in Appendix D, the total parameter account can be expressed as N(d)=65.84375d2+7438.5625d+1,880,159N(d)=65.84375d^2 +7438.5625d+1,880,159. Following the reviewer's suggestion, we repeat the fits and held out evaluations using Nadj=N−1,880,159N_{\mathrm{adj}}=N-1,880,159, essentially subtracting the width-independent term. Interestingly, despite this being an algebraic rather than necessarily physically meaningful decomposition (i.e those 1.88M params aren’t necessarily redundant), the results change quite substantially. Under the original bounds of [0.1, 2], the fitted α\alpha falls from 2 (boundary constrained) to α=1.112\alpha=1.112, and β\beta falls to 0.3820.382. Furthermore, these values remain unchanged when we relax the bounds to [0.01, 5].

\text{Parameter count} & \text{Bounds} & \alpha & \beta & \text{Lineage RMSE} & \text{Size RMSE} \\\\ \hline \text{Total }N\text{ (paper)} & [0.1,2] & 2.000^* & 0.855 & 0.0262 & 0.0518 \\\\ \text{Total }N & [0.01,5] & 2.675 & 0.994 & 0.0181 & 0.0487 \\\\ \text{Adjusted }N & [0.1,2] & 1.112 & 0.382 & 0.0180 & 0.0325 \\\\ \text{Adjusted }N & [0.01,5] & 1.112 & 0.382 & 0.0180 & 0.0325 \end{array}$$ *Upper fitting bound reached. RMSE values are in nats/token. We also repeat the same bootstrap procedure as in the paper, and whilst originally $\alpha$ hits the upper bound in 100% of the bootstrap fits, under the adjusted parameterisation, $\alpha$ doesn’t hit either bound in any draw. The final 25% bootstrap median of $\alpha$ is 1.114, with a 95% interval of [0.969, 1.299], and the endpoint results are similarly consistent at $1.108$ [0.948, 1.302]. The held-out predictions also improve: leave-lineage-out RMSE decreases from 0.0262 to 0.0180; leave-size-out RMSE decreases from 0.0518 to 0.0325 (a 37% decrease). Thus, this adjustment to N does appear to remove much of the apparent instability from $\alpha$. That being said, $\beta$ remains relatively uncertain (final-quarter bootstrap interval [0.192,0.764]), and the smallest model still accounts for ~95% of the remaining leave-size-out squared error. Moreover, the diminishing returns at larger model sizes remain: at fixed training exposure, the adjusted fit predicts an improvement of only 0.0048 nats/token between 30M and 293M parameters, compared with 0.0021 nats/token under the original fit. So whilst the fitted exponent changes quite substantially, the broader observation of relatively limited gains from further scaling in this regime remains. We thank the reviewer for suggesting this analysis, as it adds a useful nuance to our findings: the choice of parameterisation can substantially affect the fitted exponents, even when the qualitative scaling behaviour remains largely unchanged. Q2: The training windows were shuffled globally across the entire 2022-2025 period, rather than processed chronologically. Our dataloader constructs a single dataset containing all stocks and trading days, before randomly permuting the windows across the entire pool. Thus, even the smallest models sample from the full four-year period, rather than progressing through the dataset chronologically. To illustrate/confirm this, we reconstruct the sampling order for seed 42 to investigate this more explicitly. The smallest saved prefix exceeding the 17.7B-token budget contains 22.4B tokens (~0.5% of the full dataset), yet already covers all 48 months, 488 stocks and 998 trading days. Therefore, whilst the smallest and largest models received substantially fewer training tokens, these were sampled from the same underlying temporal distribution as the larger-budget runs. Thus, we can rule out the specific concern that the U-shaped endpoint CE is caused by these models having been exposed predominantly to older market data.
Reviewer_FR2TOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the scaling relationship between the size and training compute for state-space models (Mamba-3) in modeling the dynamics of limit order books in financial markets. The paper examines cross-entropy (CE) for S&P 500 order flow, examining both logged training CE from 2022–2025 and forward-time test CE on January 2026 data. The paper analyses 32 training lineages across 12 model sizes, spanning over two orders of magnitude (2.63–293.28 million parameters), with 249 January checkpoints evaluated on the same 487 tickers. Other than standard analysis on the empirical results, the paper conducts scaling law analysis on hardware-cost estimates, comparing IsoFLOP quadratics at fixed compute budgets. The results show how the additive law describes order-flow prediction and where its fitted exponents depend on the experiment and fitting choices.

Critical strengths

If presented in a way that prioritizes the benefits of human readers, an observational study on this topic with broader empirical results could have value.

Critical weaknesses

This is most likely a 100% AI-generated manuscript. The style of writing shows the characteristic style of AI: unnatural language with obscure phrases; numerics and hyperparameter configurations are extensive throughout the text in the paper, little or no elaboration on key conceptual designs or insights, monotonic presentation and figures, and the very loose layout.

The paper conducts an observational study testing existing models and checkpoints in a very narrow setting. With the human-unfriendly writing and presentation, it is unlikely to provide meaningful benefit to the readers or community.

Questions for authors

How do the authors view this manuscript's contribution to the research community?

1 comment

As detailed in our disclosure, we used AI to assist with analysis and to refine the text. We’ve been entirely transparent about this. We find it deeply concerning that the reviewer has extrapolated from a criticism of the writing style to an allegation of “little or no human effort”, alongside an ethics flag, without providing any evidence to support such a claim. Furthermore, the reviewer has assigned a soundness score of 1 without identifying a single technical flaw in the paper, or meaningfully engaging with any of the technical results.

Whilst we accept the comment that “numerics and hyperparameter configurations are extensive”, and more generally that the paper is dense, this was a deliberate choice: scaling laws are notoriously sensitive to experimental setup, fitting methodology, learning rates and schedules, etc. and prior work has demonstrated how seemingly minor methodological choices can substantially alter the resulting predictions (https://arxiv.org/abs/2404.10102, https://arxiv.org/abs/2406.19146 ). Therefore, we felt it was important to be as exhaustive as possible about our experimental configurations, rather than simply presenting a fitted scaling law without sufficient detail to scrutinise it. However, we do accept that the conceptual implications could be communicated in a clearer manner.

Regarding the supposed lack of insights, the point of the paper isn’t to introduce a new architecture or training method. It’s to understand how these models scale, and, perhaps more importantly, how reliable the resulting scaling predictions actually are. We find a rather interesting regime in which loss improves extremely rapidly at small model sizes before showing diminishing and even non-monotonic improvements under the training budgets considered. Furthermore, the scaling exponents themselves can change dramatically depending on which model sizes are included and how the laws are fitted. This is in fact precisely why we believe the paper is useful: it provides both empirical scaling results in a relatively unexplored domain and illustrates how misleading it can be to take fitted scaling exponents at face value. We also examine how these improvements in cross-entropy relate to the actual quality of generated market simulation, which is ultimately what we care about in this setting.

Lastly, we want to correct the reviewer’s description of this as a study of “existing models and checkpoints”. Every model in these scaling sweeps was trained by us for this research. The phrase in our AI disclosure, “the analysis uses existing training and evaluation results”, simply refers to the fact that the scaling runs had already been conducted when we used AI coding agents to assist with the subsequent statistical analysis. Indeed, the disclosure itself was drafted with AI assistance in an effort to be maximally transparent and truthful about its involvement. It was never intended to suggest that the models or experiments were pre-existing work.

We are perfectly happy to accept criticism of the presentation, or indeed of the scientific contribution. What we find unacceptable is the leap from disliking the writing style to making allegations about the integrity and provenance of the research, particularly when the review offers no substantive criticism of the methodology or results. We would appreciate the reviewer engaging with the actual scientific content of the paper, and identifying the specific ‘soundness’ concerns that justify the assigned scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.