- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces LOBRE, a pretrained Transformer encoder for reusable limit order book representations. LOBRE operates directly on Level-3 message streams using one token per event, with categorical and continuous price, volume, and timing information. It uses continuous-time RoPE and combines Masked Message Modeling with Aggregate Activity Forecasting, which trains the pooled sequence representation to predict future market activity. Small task-specific classifiers are then trained with the encoder either frozen or fine-tuned. The experiments use approximately 1.99 billion Nasdaq messages from 12 equities. Six assets are used for pretraining and six are excluded from pretraining. The representation is evaluated on four forecasting tasks and labeled-data budgets of 1, 3, 8, and 15 trading days. Compared with the same architecture trained from scratch, frozen pretrained LOBRE improves Macro-F1 in 90/96 configurations on pretraining assets and 88/96 configurations on held-out assets. It also outperforms the reported full fine-tuning procedure, DeepLOB, and TABL on average.
Critical strengths
- Relevant problem and sensible method. Learning a reusable representation from Level-3 message streams is useful for new forecasting tasks with limited labeled data. The one-event-per-token representation and continuous-time encoding are well matched to irregular LOB data.
- Strong controlled evaluation. The comparison with the identical randomly initialized architecture isolates the effect of pretraining. The experiments also use chronological splits, multiple labeled-data budgets, and assets completely excluded from pretraining.
- Consistent empirical results. Pretraining improves performance across most task, budget, and asset configurations. The frozen encoder remains effective with only a small task-specific classifier, supporting the practical value of reusable representations.
Critical weaknesses
- The downstream tasks are closely aligned with the Aggregate Activity Forecasting targets. Aggregate Activity Forecasting predicts future bid/ask volume changes, quote movements, elapsed time, and execution counts. These quantities closely correspond to the four downstream tasks: order-flow imbalance, mid-price movement, spread crossing, and message-frequency change. Although the downstream labels are not identical to the pretraining targets, the experiments do not establish transfer to a substantially different type of task. This weakens the paper’s central claim of a broadly reusable representation.
- The Aggregate Activity Forecasting ablation is insufficient. It is reported only for mid-price prediction, on pretraining assets, with one and three labeled days. In these settings, Masked Message Modeling alone provides little improvement over random initialization, while adding Aggregate Activity Forecasting produces most of the gain. The paper does not establish whether Aggregate Activity Forecasting improves all tasks and held-out assets, or whether its gains mainly arise when its targets closely match the downstream label.
- The evaluation covers only Nasdaq equities over a short period, with a ten-day test set. The held-out-asset setting uses target-asset labels and is therefore few-shot rather than zero-shot transfer.
Questions for authors
- Alignment between pretraining and downstream targets. How should the transfer claim be interpreted when Aggregate Activity Forecasting predicts the same underlying future quantities used to construct all four downstream labels? Can the authors provide evidence on a downstream task not directly represented by the Aggregate Activity Forecasting targets, or an ablation that removes the matching future-activity feature for each downstream task?
- Aggregate Activity Forecasting ablation. Can the authors report Masked Message Modeling alone versus the complete method across all four tasks, all data budgets, and held-out assets? This would help determine whether Aggregate Activity Forecasting learns a generally useful representation or mainly improves closely aligned tasks.
- Experimental variability. How many pretraining and downstream random seeds were used? Can the authors report variability across seeds or confidence intervals using appropriate temporal or asset-level units?
- Full fine-tuning. Were separate encoder learning rates, stronger regularization, layer-wise learning rates, or parameter-efficient adaptation methods considered? The large failures in some full fine-tuning configurations suggest that the reported comparison may depend strongly on optimization choices.
2 comments
(Continuing from previous comment.)
Seeds and variability. The primary checkpoint and each objective control come from one pretraining run, and the Limitations now state that pretraining-seed variance is unmeasured (Sec. 6). On the main assets, the frozen, fine-tuned and scratch arms have two downstream seeds. The second seed reproduces the gap over scratch training at both budgets ( at , at , 6/6 assets at ). The appendix reports the seed-1 minus seed-0 difference for each arm (Sec. 5.3, 5.5, App. A.11).
Every comparison reports a 95% bootstrap interval that resamples assets and test days, plus two-sided sign and Wilcoxon tests over assets, instead of cell win counts (Sec. 4.5, App. A.10).
Full fine-tuning. The encoder now has its own learning rate. We selected the fine-tuning recipe on validation loss among four variants: encoder rates with a head rate of , and LP-FT (Kumar et al., 2022), a linear probe followed by full fine-tuning. The four are within 0.003 of each other in validation loss (App. A.2, A.7).
The large fine-tuning failures came from the submitted shared schedule. Without the collapsed runs, the submitted frozen advantage falls from to (App. A.14). With the tuned recipe, fine-tuning edges the frozen head by over two seeds (CI ), and we withdraw the claim that freezing beats fine-tuning (Sec. 5.5).
Parameter-efficient methods sit between the two arms we evaluate: the frozen base with a 0.13M-parameter head, and full fine-tuning. Since these two are close, the case for freezing now rests on reuse (one base serving every task) rather than on accuracy (Sec. 5.5).
Thank you for the review and the feedback. Please find our responses below to the main weaknesses and concerns you had raised.
In the revision, every comparator's learning rate is selected on validation loss (Sec. 4.5, App. A.7).
Task alignment with AAF. Table 1 now lists each task's closest AAF target and how much it overlaps with the label (Sec. 4.2). If AAF's gain came from previewing the labels, it should be largest on the high-overlap tasks. It is not (Sec. 6, "Overlap between AAF targets and downstream labels"):
Regarding removing the matching AAF target for each task, that ablation needs a separately pretrained encoder for every removed target. Each such encoder also has less pretraining signal and a different loss balance, the confounds our matched MMM-only and AAF-only controls avoid. We therefore test the prediction such an ablation would check, that the gain tracks label overlap, across all four tasks with matched controls. A task decoupled from short-horizon order flow remains untested, and the revised claims are narrowed accordingly to short-horizon order-flow forecasting (Sec. 6, 7).
Scope of the AAF ablation. The ablation now covers all four tasks at B∈{1,15} (48 cells per arm). The MMM-only encoder is retrained on the same data and schedule as the primary checkpoint, replacing the submitted one. AAF improves every task–budget cell: +0.055 (CI [0.040,0.069], 6/6 assets) (Sec. 4.4, 5.2; Fig. 1). This also answers whether AAF helps only closely aligned tasks: its largest gain is on the least-aligned task (see above). We also added two controls:
The objective controls were run on the main assets, where all arms are pretrained on identical data. On the held-out assets, the primary checkpoint gains as much over tuned scratch training (+0.048) as on the main assets (+0.050) (Sec. 5.4). The Limitations state that the objective controls cover the main assets only (Sec. 6).
Data scope, few-shot transfer. We add six far assets from other sectors, excluded from pretraining and tested in a second ten-day period about six months later (Sec. 4.1, 5.4, App. A.12). The gain over scratch training extends to them (+0.049 Macro-F1, CI [0.033, 0.067], 6/6 assets). On these assets frozen LOBRE is slightly below both feature models: −0.015 against gradient-boosted trees (CI [−0.027,0.001], not significant) and −0.018 against logistic regression (CI [−0.032,−0.0003]). A post-hoc check finds the gain unchanged on test windows with top-decile volatility, message rate or spread: +0.051 vs +0.050, 12/12 assets (Sec. 6, App. A.13). Other exchanges and asset classes are outside our scope (Sec. 6).
The held-out setting is few-shot by design: the question is how many target labels a new task needs, and each task requires a trained head. The paper frames the held-out and far evaluations as low-label adaptation with one or fifteen days of target labels (Sec. 1, 4).
(Continued in the next comment.)