acceptodds
Under review as a conference paper at ICLR 2027

LOBRE: Transferable Limit Order Book Representations for Low-Data Regimes

Abstract

Limit order book forecasting models are typically trained for individual prediction tasks, making it difficult to rapidly develop models for applications with limited task-specific labels, such as forecasting within infrequent Limit Order Book (LOB) market regimes. We introduce LOBRE, a transformer encoder pretrained directly on irregular Level-3 message streams to learn reusable representations for downstream forecasting with scarce labels. LOBRE combines Masked Message Modeling (MMM) with Aggregate Activity Forecasting (AAF), a prospective objective that trains a pooled sequence representation to capture information predictive of subsequent market activity. We evaluate transfer across heterogeneous forecasting tasks, multiple target-data budgets, and assets both included in and excluded from pretraining. With the encoder frozen, pretraining improves Macro-F1 over the same architecture trained from scratch in 90/96 configurations on pretraining assets and 88/96 configurations on held-out assets, with mean gains of 0.092 and 0.097, respectively. Frozen adaptation also outperforms full fine-tuning on average and exceeds DeepLOB and TABL benchmarks across all evaluated task–budget comparisons. These results support a pretrain-once, adapt-many approach to limit order book forecasting, in which a shared message-level representation can be reused for new tasks and assets with limited target-specific supervision.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised pretraining, financial time series, representation learning, transfer learning, limit order book

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_C7VKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LOBRE, a pretrained Transformer encoder for reusable limit order book representations. LOBRE operates directly on Level-3 message streams using one token per event, with categorical and continuous price, volume, and timing information. It uses continuous-time RoPE and combines Masked Message Modeling with Aggregate Activity Forecasting, which trains the pooled sequence representation to predict future market activity. Small task-specific classifiers are then trained with the encoder either frozen or fine-tuned. The experiments use approximately 1.99 billion Nasdaq messages from 12 equities. Six assets are used for pretraining and six are excluded from pretraining. The representation is evaluated on four forecasting tasks and labeled-data budgets of 1, 3, 8, and 15 trading days. Compared with the same architecture trained from scratch, frozen pretrained LOBRE improves Macro-F1 in 90/96 configurations on pretraining assets and 88/96 configurations on held-out assets. It also outperforms the reported full fine-tuning procedure, DeepLOB, and TABL on average.

Critical strengths
  1. Relevant problem and sensible method. Learning a reusable representation from Level-3 message streams is useful for new forecasting tasks with limited labeled data. The one-event-per-token representation and continuous-time encoding are well matched to irregular LOB data.
  2. Strong controlled evaluation. The comparison with the identical randomly initialized architecture isolates the effect of pretraining. The experiments also use chronological splits, multiple labeled-data budgets, and assets completely excluded from pretraining.
  3. Consistent empirical results. Pretraining improves performance across most task, budget, and asset configurations. The frozen encoder remains effective with only a small task-specific classifier, supporting the practical value of reusable representations.
Critical weaknesses
  1. The downstream tasks are closely aligned with the Aggregate Activity Forecasting targets. Aggregate Activity Forecasting predicts future bid/ask volume changes, quote movements, elapsed time, and execution counts. These quantities closely correspond to the four downstream tasks: order-flow imbalance, mid-price movement, spread crossing, and message-frequency change. Although the downstream labels are not identical to the pretraining targets, the experiments do not establish transfer to a substantially different type of task. This weakens the paper’s central claim of a broadly reusable representation.
  2. The Aggregate Activity Forecasting ablation is insufficient. It is reported only for mid-price prediction, on pretraining assets, with one and three labeled days. In these settings, Masked Message Modeling alone provides little improvement over random initialization, while adding Aggregate Activity Forecasting produces most of the gain. The paper does not establish whether Aggregate Activity Forecasting improves all tasks and held-out assets, or whether its gains mainly arise when its targets closely match the downstream label.
  3. The evaluation covers only Nasdaq equities over a short period, with a ten-day test set. The held-out-asset setting uses target-asset labels and is therefore few-shot rather than zero-shot transfer.
Questions for authors
  1. Alignment between pretraining and downstream targets. How should the transfer claim be interpreted when Aggregate Activity Forecasting predicts the same underlying future quantities used to construct all four downstream labels? Can the authors provide evidence on a downstream task not directly represented by the Aggregate Activity Forecasting targets, or an ablation that removes the matching future-activity feature for each downstream task?
  2. Aggregate Activity Forecasting ablation. Can the authors report Masked Message Modeling alone versus the complete method across all four tasks, all data budgets, and held-out assets? This would help determine whether Aggregate Activity Forecasting learns a generally useful representation or mainly improves closely aligned tasks.
  3. Experimental variability. How many pretraining and downstream random seeds were used? Can the authors report variability across seeds or confidence intervals using appropriate temporal or asset-level units?
  4. Full fine-tuning. Were separate encoder learning rates, stronger regularization, layer-wise learning rates, or parameter-efficient adaptation methods considered? The large failures in some full fine-tuning configurations suggest that the reported comparison may depend strongly on optimization choices.
2 comments

Thank you for the review and the feedback. Please find our responses below to the main weaknesses and concerns you had raised.

In the revision, every comparator's learning rate is selected on validation loss (Sec. 4.5, App. A.7).

Task alignment with AAF. Table 1 now lists each task's closest AAF target and how much it overlaps with the label (Sec. 4.2). If AAF's gain came from previewing the labels, it should be largest on the high-overlap tasks. It is not (Sec. 6, "Overlap between AAF targets and downstream labels"):

  • Over MMM-only, the frozen gain is largest on mid-price, the lowest-overlap task: +0.108+0.108 (CI [0.076,0.133][0.076, 0.133]), above the other three tasks' intervals.
  • It is smallest on message frequency (+0.028+0.028), a high-overlap task.
  • With the readouts equalized, the gain from the representation alone is again largest on mid-price at B=1B=1 (+0.110+0.110).
  • The gain does not shrink as target labels grow (+0.054+0.054 to +0.056+0.056 across budgets).

Regarding removing the matching AAF target for each task, that ablation needs a separately pretrained encoder for every removed target. Each such encoder also has less pretraining signal and a different loss balance, the confounds our matched MMM-only and AAF-only controls avoid. We therefore test the prediction such an ablation would check, that the gain tracks label overlap, across all four tasks with matched controls. A task decoupled from short-horizon order flow remains untested, and the revised claims are narrowed accordingly to short-horizon order-flow forecasting (Sec. 6, 7).

Scope of the AAF ablation. The ablation now covers all four tasks at B∈{1,15}B \in \{1, 15\} (48 cells per arm). The MMM-only encoder is retrained on the same data and schedule as the primary checkpoint, replacing the submitted one. AAF improves every task–budget cell: +0.055+0.055 (CI [0.040,0.069][0.040, 0.069], 6/6 assets) (Sec. 4.4, 5.2; Fig. 1). This also answers whether AAF helps only closely aligned tasks: its largest gain is on the least-aligned task (see above). We also added two controls:

  • Readout-equalized comparison. With both encoders' readouts retrained on the labels, AAF still leads by +0.039+0.039 (6/6 assets), so it changes the representation itself. This meets a criterion we fixed before reading the MMM+AAF results of this control.
  • AAF-only encoder. It is +0.017+0.017 above MMM+AAF on all six assets (Sec. 5.2).

The objective controls were run on the main assets, where all arms are pretrained on identical data. On the held-out assets, the primary checkpoint gains as much over tuned scratch training (+0.048+0.048) as on the main assets (+0.050+0.050) (Sec. 5.4). The Limitations state that the objective controls cover the main assets only (Sec. 6).

Data scope, few-shot transfer. We add six far assets from other sectors, excluded from pretraining and tested in a second ten-day period about six months later (Sec. 4.1, 5.4, App. A.12). The gain over scratch training extends to them (+0.049+0.049 Macro-F1, CI [0.033, 0.067], 6/6 assets). On these assets frozen LOBRE is slightly below both feature models: −0.015-0.015 against gradient-boosted trees (CI [−0.027,0.001][-0.027, 0.001], not significant) and −0.018-0.018 against logistic regression (CI [−0.032,−0.0003][-0.032, -0.0003]). A post-hoc check finds the gain unchanged on test windows with top-decile volatility, message rate or spread: +0.051+0.051 vs +0.050+0.050, 12/12 assets (Sec. 6, App. A.13). Other exchanges and asset classes are outside our scope (Sec. 6).

The held-out setting is few-shot by design: the question is how many target labels a new task needs, and each task requires a trained head. The paper frames the held-out and far evaluations as low-label adaptation with one or fifteen days of target labels (Sec. 1, 4).

(Continued in the next comment.)

(Continuing from previous comment.)

Seeds and variability. The primary checkpoint and each objective control come from one pretraining run, and the Limitations now state that pretraining-seed variance is unmeasured (Sec. 6). On the main assets, the frozen, fine-tuned and scratch arms have two downstream seeds. The second seed reproduces the gap over scratch training at both budgets (+0.067+0.067 at B=1B=1, +0.021+0.021 at B=15B=15, 6/6 assets at B=15B=15). The appendix reports the seed-1 minus seed-0 difference for each arm (Sec. 5.3, 5.5, App. A.11).

Every comparison reports a 95% bootstrap interval that resamples assets and test days, plus two-sided sign and Wilcoxon tests over assets, instead of cell win counts (Sec. 4.5, App. A.10).

Full fine-tuning. The encoder now has its own learning rate. We selected the fine-tuning recipe on validation loss among four variants: encoder rates {10−5,3×10−5,10−4}\{10^{-5}, 3\times10^{-5}, 10^{-4}\} with a head rate of 10−310^{-3}, and LP-FT (Kumar et al., 2022), a linear probe followed by full fine-tuning. The four are within 0.003 of each other in validation loss (App. A.2, A.7).

The large fine-tuning failures came from the submitted shared schedule. Without the collapsed runs, the submitted frozen advantage falls from +0.041+0.041 to +0.025+0.025 (App. A.14). With the tuned recipe, fine-tuning edges the frozen head by +0.006+0.006 over two seeds (CI [0.0004,0.014][0.0004, 0.014]), and we withdraw the claim that freezing beats fine-tuning (Sec. 5.5).

Parameter-efficient methods sit between the two arms we evaluate: the frozen base with a 0.13M-parameter head, and full fine-tuning. Since these two are close, the case for freezing now rests on reuse (one base serving every task) rather than on accuracy (Sec. 5.5).

Reviewer_HVWXOpenReview ↗
Rating
3
Confidence
2
Informativeness
3(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper presents LOBRE, a small bidirectional transformer encoder pretrained on Nasdaq Level-3 message streams. Each market message is one sequence position, which combines a categorical token with continuous price, volume and inter-arrival features, and irregular timing is handled with a continuous-time RoPE. Pretraining combines Masked Message Modeling (MMM) with Aggregate Activity Forecasting (AAF), which trains the pooled representation zz to predict aggregate activity at offsets of 25, 50 and 100 messages. The encoder is then reused and kept frozen, with lightweight heads for four short-horizon classification tasks. Pretrained-frozen LOBRE improves Macro-F1 in most paired configurations, with mean gains of about 0.09 in both asset groups. Frozen adaptation performs better than full fine-tuning on average under the reported setup, and LOBRE exceeds DeepLOB and TABL in the evaluated comparisons. A limited mid-price ablation suggests that AAF contributes substantially to the benefit.

Critical strengths
  • The experimental design is reasonably well controlled. The LOBRE variants share the same architecture, with paired comparisons on the same data windows, chronological splits, nested budgets, and assets fully excluded from pretraining, this makes the main comparison straightforward to assess. The held-out gain is close to the main-asset gain (0.097 vs 0.092), and this is convincing evidence of cross-asset transfer within Nasdaq equities.
  • Modeling the message stream with one position per event and reusing a frozen pooled representation is a sensible practical direction. It complements snapshot-based representation work such as SimLOB and LOBench, as well as generative order-flow models. The appendix also provides substantial implementation detail, it is detailed enough that the pipeline should be easily reproducible in principle.
Critical weaknesses
  • Some comparators perform poorly, which makes the size of the reported gains harder to interpret. For example, at B=1B=1, random initialization reaches about 0.44 and 0.43 (main and heldout) Macro-F1 on τmidprice\tau_{\text{midprice}}; On this near-balanced test set, this is below the roughly 0.5 which a uniform-guesser would get. Some full fine-tuning runs also seem to collapse. For example, GOOG τmidprice\tau_{\text{midprice}} at B=8B=8 unfrozen gets 0.333 (and GOOG τfreq\tau_{\text{freq}} at B=8B=8 gets 0.144), which somewhat behaves like a constant predictor. A possible reason might be that all downstream variants share one schedule with a peak learning rate of 5×10−45\times10^{-4}. That is probably fine for a frozen head, but may not suit full fine-tuning or training a 2.35M-parameter model from scratch on a few thousand samples.
  • Seeds and variance are not reported; also the 90/96 win counts are not independent since they share one encoder and the same test days. On some cells the absolute performance stays close to chance even with pretraining, for example, when τofi\tau_{\text{ofi}} at B=1B=1: 0.346.
  • Also a minor issue, the model is called LOBRE in the main text but LOBERT in the appendix.
Questions for authors
  • During MMM, is the RoPE coordinate Ti=∑j=1iΔt~jT_i=\sum_{j=1}^{i}\widetilde{\Delta t}_j computed from the original or corrupted inter-arrival times? It is stated that these values are masked in the appendix, so please clarify how this interacts with the time loss.
  • Could you report supervised mid-price prediction results using all 75 training days per asset? This would help compare low-budget adaptation with a model trained on more target data.
  • Were learning rates tuned separately for random initialization, full fine-tuning, DeepLOB and TABL? In my opinion, the full fine-tuning result with a smaller rate would help confirm the frozen better than unfrozen claim.
1 comment

Thank you for the insightful review. Please find below our responses to the primary weaknesses and concerns raised in your feedback.

The evaluation was re-run with a separately selected learning rate for every arm.

Weak comparators, separately tuned learning rates. Every arm now has its own learning rate, selected on validation loss: scratch training, fine-tuning, DeepLOB, TABL and the frozen heads. App. A.7 gives each grid, the selection rule and the selected values.

  • Scratch training. It is still below chance on mid-price at B=1B=1 (0.454 vs. 0.500, now shown as a uniform-guess row in Table 2), but it reaches 0.625 at B=15B=15. The recipe therefore works and is limited by labels at B=1B=1 (Sec. 5.3). Frozen LOBRE gains +0.049+0.049 over tuned scratch training on all twelve assets. Note that Macro-F1 scores can be below uniform guessing if the model's outputs are skewed towards one class.
  • Fine-tuning. The base uses a smaller learning rate (10−510^{-5}, head 10−310^{-3}), selected among four variants including LP-FT. It now edges the frozen head by +0.006+0.006 over two seeds (CI [0.0004,0.014][0.0004, 0.014]). We withdraw the claim that freezing beats fine-tuning (Sec. 5.5).
  • Effect of collapsed runs. Re-scoring the submitted runs reproduces the frozen result (0.445 vs 0.446). The submitted frozen advantage over fine-tuning falls from +0.041+0.041 to +0.025+0.025 without the collapsed runs (App. A.14).

Seeds and variance. We replaced cell win counts with two-sided sign and Wilcoxon tests over assets and with 95% bootstrap intervals that resample assets and test days. Together these account for the shared encoder and shared test days (Sec. 4.5).

On the main assets, we ran a second downstream seed for the frozen, fine-tuned and scratch arms. It reproduces the gap over scratch training at both budgets: +0.067+0.067 at B=1B=1 and +0.021+0.021 at B=15B=15 (CI [0.013,0.030][0.013, 0.030], 6/6 assets; Sec. 5.3, 5.5, App. A.11). The Limitations state that pretraining-seed variance is unmeasured (Sec. 6).

For near-chance cells such as OFI at B=1B=1, the new feature baselines show what is attainable. Logistic regression leads there (0.363 vs 0.345), using a feature close to the OFI label. Overall, frozen LOBRE matches or exceeds both feature models on the main and held-out assets (Sec. 5.1, 5.4).

LOBRE/LOBERT naming. Fixed: the appendix now uses LOBRE throughout. LOBERT was a remnant from an earlier version.

RoPE under masking. The RoPE coordinate TiT_i is computed from the model input. Corrupted positions therefore use the corrupted inter-arrival time: 0.50.5 for masked messages and a draw from U(0,1)\mathcal{U}(0,1) for random replacements. The time loss targets the original value, so a masked Δt~i\widetilde{\Delta t}_i cannot be recovered from Ti−Ti−1T_i - T_{i-1} (App. A.2, "Corruption and temporal position").

Training on all 75 days. We've added a note in the Limitations (Sec. 6) that this is out of scope of this study. As a reference at the same budgets, the revision adds feature models and tuned end-to-end models (Sec. 5.1).

Reviewer_pUESOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LOBRE learns representations of limit order book messages through masked-message reconstruction (MMM) and future-activity prediction (AAF). A frozen encoder then supports four forecasting tasks with limited target-specific data. The reported gains are promising, including on stocks excluded from pretraining. However, I find the contribution primarily an incremental application of established techniques.

Critical strengths

Strengths

  • The paper addresses an interesting practical question: whether one pretrained representation can support several market-forecasting tasks with limited target data.

  • Pretraining improves performance across most evaluated configurations, including held-out stocks.

  • The evaluation considers multiple tasks, training budgets, and chronological testing. Frozen adaptation also offers practical reuse of the encoder.

Critical weaknesses

Weaknesses

  • Limited innovation. The method combines familiar components without clearly establishing a substantial methodological advance or a broader scientific insight. This is my principal concern.

  • Incomplete evidence for the proposed combination. There is no AAF-only baseline. Moreover, the described MMM-only procedure appears to leave its learned pooler without pretraining supervision before freezing it, complicating the attribution of gains to AAF.

  • Limited comparative evidence. DeepLOB and TABL are useful forecasting baselines, but neither receives comparable pretraining. The results establish benefits over scratch training more clearly than advantages over alternative representation-learning methods.

  • Narrow demonstrated transfer. The experiments cover one exchange and a limited period, with downstream targets closely related to AAF. Transfer to rare regimes or substantially different objectives remains untested.

Questions for authors

Questions

  • What new methodological insight does the proposed combination establish beyond the effectiveness of applying existing pretraining techniques to market messages?

  • Does MMM-only pretraining update the pooler? What evidence distinguishes the benefit of combining MMM and AAF from AAF alone or differences in pooling supervision?

  • How does the method compare with alternative pretrained representations under comparable data and adaptation budgets?

  • What evidence supports transfer beyond objectives closely related to AAF, and how narrowly should the generalization claims be interpreted?

2 comments

Thank you for the review and the helpful feedback. Please find our responses below to the main improvement areas you had mentioned.

Limited innovation, methodological insight. The revision states the insight explicitly and tests it with controls matched in data and schedule:

  • Masked modeling never trains the pooled representation that a frozen head reads.
  • A frozen encoder pretrained with MMM alone is therefore no better than training from scratch: −0.005-0.005 (CI [−0.012,0.002][-0.012, 0.002]).
  • Even with a readout trained on the labels, the MMM-only encoder stays below hand-crafted features on all six assets.
  • Adding AAF raises the frozen encoder by +0.055+0.055 Macro-F1, to the level of those features.

The resulting design rule: an encoder meant for frozen reuse should be pretrained with a loss on the representation its heads read. For frozen reuse, the pretraining objective matters more than the decision to pretrain (Sec. 1, 5.2, 6; Fig. 1).

AAF-only baseline, MMM-only pooler. MMM-only pretraining does not update the pooler. It receives no gradient and stays at its initialization, which the paper now states (Sec. 3.3). We added two controls (Sec. 4.4, 5.2, App. A.8):

  • Pooling supervision vs. representation. We reset the readouts (pooler and layer mix) of both encoders and trained them on the labels, keeping the encoder bodies frozen. MMM+AAF still leads by +0.039+0.039 (CI [0.033,0.046][0.033, 0.046], 6/6 assets). This meets a criterion we fixed before reading the MMM+AAF results of this control, so AAF changes the representation itself, not only the pooler.
  • AAF alone. An AAF-only encoder with matched data and schedule is +0.017+0.017 above MMM+AAF and +0.072+0.072 above MMM-only (6/6 assets each). When frozen, it exceeds both feature models. The transferable signal therefore comes from AAF; MMM adds nothing measurable on top of it. AAF-only comes from a single pretraining run, as the Limitations state (Sec. 6).

Comparison with pretrained representations. The pretrained-representation baseline is now the same encoder pretrained with MMM on identical data and schedule (Sec. 4.3). DeepLOB and TABL each have their own tuned learning rate. We also add gradient-boosted trees and logistic regression on 28 microstructure features, which no tuned end-to-end deep model reaches (Sec. 5.1). Snapshot-level pretrained models such as SimLOB and LOBench operate on reconstructed book states rather than message streams. We did not compare against them, as the Related Work and Limitations now state (Sec. 2, 6).

Narrow transfer, objectives close to AAF. Table 1 now lists each task's closest AAF target and how much it overlaps with the label (Sec. 4.2). AAF's advantage does not follow that overlap. It is largest on mid-price, the lowest-overlap task (+0.108+0.108), and does not shrink with more labels (Sec. 6, "Overlap between AAF targets and downstream labels").

We have narrowed the generalization claims to short-horizon order-flow forecasting on large-cap equities in Nasdaq order-book data (Sec. 6, 7). Within that scope, the gain extends to six far assets from other sectors, tested about six months later (Sec. 5.4).

Thank you for the detailed response and the new controls, especially the AAF-only ablation and the retrained-readout comparison, which answer my second question. However, my main concern remains the limited substantial innovation, so I will keep my score.

Reviewer_yKXyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose a small (2.35M-parameter) bidirectional transformer encoder pretrained on Nasdaq Level-3 message streams. Each message becomes one token. The token combines a categorical code with continuous price, volume, and timing features. Continuous-time RoPE handles irregular message arrivals. The pretraining uses two objective, Masked Message Modeling (MMM) and Aggregate Activity Forecasting (AAF). MMM reconstructs hidden messages. AAF trains the pooled sequence vector to predict aggregate order flow 25 to 100 messages ahead. The authors pretrain on six stocks. They then train lightweight heads for four short-horizon classification tasks with 1 to 15 days of labels. They test on the six pretraining stocks and on six unseen stocks. The frozen encoder beats the same model trained from scratch in over 90% of configurations, with a gain of about 0.09 Macro-F1. The frozen encoder also beats full fine-tuning on average. It outperforms DeepLOB and TABL as well.

Critical strengths

This paper provide a clear evidence that one message-level encoder transfers across stocks. The pretrained and scratch models share the same architecture, data, and labels. Only the initialization differs, so the gain comes from pretraining itself. On six stocks never seen in pretraining, the gain (about 0.10 Macro-F1) matches the gain on the pretraining stocks.

The proposed AAF ideaa fills a real gap in BERT-style pretraining for forecasting. Masked message modeling trains each token’s state but never trains the pooled vector that downstream heads use. AAF fixes this by training the pooled vector to predict future market activity. The idea is simple and fits the forecasting use case well.

Critical weaknesses

The data scope is too narrow for a “pretrain once, adapt many” claim. The paper uses twelve large Nasdaq stocks from one exchange over about four months. The unseen stocks resemble the pretraining stocks closely, and on mid-price they even score higher. The rare-regime setting that motivates the paper is never tested, since the label budgets are simply the most recent days. The paper also does not vary pretraining data size or model size, so it remains unclear whether more pretraining helps.

The baselines are too weak, so the gains look bigger than they are. On two of the four tasks, the model trained from scratch does no better than random guessing at any label budget. DeepLOB does worse than random guessing on every task. If pretraining truly saved labels, its advantage should shrink as more labels become available. Instead, the advantage stays almost the same from 1 to 15 days of labels. This suggests the scratch model simply fails to train, rather than lacking data. The claim that freezing beats fine-tuning has the same problem. The biggest gaps come from fine-tuned runs that broke down and predicted only one class.

Questions for authors

How does LOBRE perform on stocks that differ more from the pretraining set, such as small-cap stocks, large-tick stocks, or stocks from another exchange? Why does DeepLOB score below random guessing on every task? At what label budget does the scratch model catch up with the pretrained model? How large is the gain when you train on the full training period?

1 comment

Thank you for the review and insightful feedback. Please find below our responses to the main weaknesses and concerns you had raised.

In the revision, every comparator's learning rate is selected on validation loss (Sec. 4.5, App. A.7). Several numbers below differ from the original submission as a result.

Weak scratch baseline, catch-up budget. Scratch training now has its own learning rate, selected on validation loss. With it, the scratch model beats a uniform guess on all four tasks at B=15B=15. On mid-price it is still below chance at B=1B=1 (0.454 vs 0.500) but reaches 0.625 at B=15B=15, above both new feature baselines. Its B=1B=1 deficit therefore reflects label scarcity, not a failure to train (Sec. 5.3, Table 2).

The gain now shrinks with more labels. From B=1B=1 to B=15B=15, it falls from +0.073+0.073 to +0.028+0.028 on the main assets and from +0.069+0.069 to +0.027+0.027 on the held-out assets, with all four intervals above zero. At B=15B=15, scratch training catches up on mid-price and message frequency, while the gap persists on OFI and spread crossing (Sec. 5.3, Fig. 2).

DeepLOB below random guessing. The submitted DeepLOB applied BatchNorm to raw price and volume levels and predicted a single class in 19 of 48 runs. The revised DeepLOB normalizes each input window, uses its original input length of 100 snapshots and has a tuned learning rate. Only 4 of 48 runs now collapse, and it beats a uniform guess on every task at B=15B=15 (App. A.5, A.14). Frozen LOBRE exceeds tuned DeepLOB and TABL by 0.055 and 0.059 (6/6 assets each). Note that Macro-F1 scores can be below uniform guessing if the model's predictions are skewed towards one class.

As a stronger non-deep reference, we add gradient-boosted trees and logistic regression on 28 hand-crafted microstructure features. Frozen LOBRE matches or exceeds them (Sec. 5.1).

Frozen vs. fine-tuned. Fine-tuning now uses a tuned recipe: base learning rate 10−510^{-5}, selected among four variants including LP-FT. With it, we withdraw the claim that freezing beats fine-tuning. Over two downstream seeds, fine-tuning edges the frozen head by +0.006+0.006 Macro-F1 (CI [0.0004,0.014][0.0004, 0.014], 5/6 assets). On the held-out assets, fine-tuning is +0.007+0.007 above the frozen head and level with both feature models. The case for freezing now rests on reuse: one 2.35M-parameter base serves all tasks, each with a 0.13M-parameter head (Sec. 5.5). In the re-scored submitted runs, the frozen advantage was +0.041+0.041 and falls to +0.025+0.025 without the collapsed runs (App. A.14).

Narrow data scope, more different stocks. We add six far assets from other sectors: energy, pharmaceuticals, biotechnology, mining, consumer staples and financials. They are excluded from pretraining and tested about six months later (Sec. 4.1). The gain over scratch training extends to them (+0.049+0.049 Macro-F1, CI [0.033, 0.067], 6/6 assets). On these assets frozen LOBRE is slightly below both feature models: −0.015-0.015 against gradient-boosted trees (CI [−0.027,0.001][-0.027, 0.001], not significant) and −0.018-0.018 against logistic regression (CI [−0.032,−0.0003][-0.032, -0.0003]) (Sec. 5.4, App. A.12). Small-cap and large-tick stocks and other exchanges are outside the revised study; the Limitations state that the data cover large-cap equities from one exchange (Sec. 6).

Rare regimes. The revision no longer claims transfer to market regimes. The model name now expands to "Representation Encoder", and the paper presents the budgets as a test of label scarcity (Sec. 1, 4). The test period is busier and more volatile than the training days for all twelve assets (Sec. 4).

A post-hoc check finds the gain unchanged on test windows that are atypical for each asset (top-decile volatility, message rate or spread). There it is +0.051+0.051 vs +0.050+0.050 on the other windows, on 12/12 assets (Sec. 6, App. A.13). We describe these windows as atypical, not as regimes.

Pretraining scale, full training period. These are outside the revised study, and the Limitations state both (Sec. 6). As a reference for what more target-side information yields at the same budgets, the revision adds the feature models and tuned end-to-end models (Sec. 5.1).

Comments on the paper

1 comment
AuthorsOpenReview ↗

To all reviewers

We thank the reviewers for their reviews and feedback. The shared main concerns were related to (among other individual concerns):

  1. undertuned comparators,
  2. attribution of the gain to AAF,
  3. variance and statistics,
  4. the scope of transfer, including how closely the downstream tasks align with AAF.

In response, we re-ran the evaluation under a protocol that selects every comparator's learning rate on validation loss (Sec. 4.5, App. A.7).

The revision is substantially rewritten, so changes are not colour-marked in the PDF. The section references in each response point to the corresponding changes.

Two headline results changed:

  • The frozen gain over tuned scratch training is now +0.049+0.049 Macro-F1 (95% CI [0.041,0.058][0.041, 0.058], 12/12 assets). The submission reported ≈0.09\approx 0.09.
  • Freezing no longer beats a tuned fine-tuning recipe, and we withdraw that claim (Sec. 5.5).

The revision adds:

  • hand-crafted feature baselines (Sec. 5.1);
  • matched MMM-only and AAF-only pretraining controls, plus a decomposition with retrained readouts (Sec. 4.4, 5.2);
  • a second downstream seed (Sec. 5.3, App. A.11);
  • six far assets from other sectors, tested about six months later (Sec. 4.1, 5.4, App. A.12);
  • a re-scoring of the submitted runs (App. A.14).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.