acceptodds
Under review as a conference paper at ICLR 2027

UltraBERT: Beyond the Limit of BERT Pre-Training

14 views

Abstract

Recent bidirectional encoders have been modernized with innovations from causal large language models (LLMs). However, the pretraining recipe itself remains inherited from small-scale conventions, leaving encoders underoptimized, with no single model performing competitively across both classification and retrieval tasks. In this work, we investigate how far encoder performance can be pushed by revisiting core design choices in training, revealing three key findings: (1) Encoders saturate on downstream tasks beyond a few trillion tokens, indicating that additional compute past this point does not help. (2) Across model sizes, lower masking ratios benefit classification while higher ones benefit retrieval, contradicting prior work suggesting larger ratios for larger models. (3) Distillation with pruning improves classification at low masking ratios but degrades retrieval at high ones. Guided by these findings, we present UltraBERT, a single encoder that assigns separate LM heads to low and high masking ratios within a batch and restricts the distillation objective to the low ratio. To our knowledge, UltraBERT is the first encoder to achieve state of the art on both classification and retrieval tasks, reaching 91.0 on GLUE and 47.7 on BEIR. Together, these results show how far encoder performance can be pushed and provide a principled basis for compute allocation, masking ratio design, and knowledge transfer in bidirectional models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
bert, bidirectional encoder, pre-training

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 4.00 over 3 reviews, above 99% of the others.
Reviewer_9tagOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This work studies the effects of pre-training recipes for encoder models at scale, and uses the findings to train a new encoder model (UltraBERT) that is shown to achieve SOTA performance on a wide range of classification and information retrieval benchmarks at both a base and medium model size. It finds that downstream performance of encoder models saturates with a large number of pre-training tokens unlike decoder models, that there's a trade-off in the masking ratio for performance on classification and retrieval tasks, and shows that pre-training using LM head decoupling can achieve the best of both worlds.

Critical strengths
  • The paper is well written and easy to follow.
  • The multiple studies on the effect of large scale training on encoders are insightful, particularly the pre-training saturation and the link between masking ratio and the effective rank of the representations.
  • Using decoupled LM heads to achieve the best of both worlds is insightful.
  • UltraBERT achieves strong performance on downstream benchmarks.
Critical weaknesses
  • Only 4 seeds are used for the small GLUE tasks, 3 for MultiCoNER (for selection), and the rest of the seed counts do not seem to be reported. Seed variances and statistics are also not reported. One of the headline results is 91.0 on GLUE with the second best model obtaining 90.8; it's not clear how far ahead UltraBERT is, without the seed variance being shown.
  • No evaluation on multi-vector ColBERT style IR tasks. The paper states that it follows the ModernBERT evaluation protocol [1] which itself tests on multi-vector IR tasks.

[1] Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Griffin Thomas Adams, Jeremy Howard, and Iacopo Poli. Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025.

Questions for authors
  • [2] tests different masking ratios for a non-transformer encoder and shows (in tables 8 and 9 of [2]) that sequence classification peaks at 20% while IR is best at 40-50%, in line with UltraBERT's finding that retrieval favors higher masking ratios than classification. Although this is done at a much smaller token budget, this could serve as useful context. Have the authors considered comparing against [2], which is also shown to achieve strong performance on IR and long context tasks?
  • Could the authors report seed SD and possibly run evals with more seeds (at least 5-10)?
  • Have the authors tested performance on multi-vector ColBERT style tasks, as [1] does? I'd be curious to see if the advantages of higher masking rates translate over to multi-vector tasks.
  • Can the authors explain their choice of the subset of benchmarks used in section 3 with dropping SQuAD in tables 1 and 2, and not including NER? Particularly, the difference between the benchmark scores is reported to be much less drastic for NER in tables 8 and 9 of [2] with different masking ratios, compared to sequence classification and IR. I am curious how much the observed effects of differences in masking ratio (and of some of the other ablations) could be benchmark dependent.

[2] Devang Acharya and Mohammad Hammoud. Avey-B. In The Fourteenth International Conference on Learning Representations, 2026.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 9tag (Part 1/3)

We thank the reviewer for the thorough review, and we are glad that the paper was found well written, and the studies on pre-training saturation and the decoupled LM heads insightful. We respond to each point below.


Only 4 seeds are used for the small GLUE tasks, 3 for MultiCoNER (for selection), and the rest of the seed counts do not seem to be reported. Seed variances and statistics are also not reported. One of the headline results is 91.0 on GLUE with the second best model obtaining 90.8; it's not clear how far ahead UltraBERT is, without the seed variance being shown.

Could the authors report seed SD and possibly run evals with more seeds (at least 5-10)?

We re-evaluate all main benchmarks with 5 seeds for our two models (UltraBERT-150M and UltraBERT-380M). We confirm that the superiority of UltraBERT remains unchanged under multi-seed evaluation.

For classification, we repeat the evaluation protocol 5 times with different seeds. For retrieval, we repeat the MS MARCO fine-tuning 5 times with different seeds and evaluate each model, and the MLDR fine-tuning continues from each of these MS MARCO models.

ClassificationGLUESQuAD 2.0MultiCoNER v2Few-NERDRewardBench
Base (≤200M)
Best Baseline88.987.460.167.170.3
UltraBERT89.589.257.967.473.0
+ 5 seeds89.34 ± 0.1789.17 ± 0.1757.93 ± 0.2867.43 ± 0.0671.96 ± 0.72
Medium (≤500M)
Best Baseline90.890.463.068.176.7
UltraBERT91.090.963.568.378.8
+ 5 seeds90.90 ± 0.1090.79 ± 0.1563.51 ± 0.1368.23 ± 0.0778.88 ± 0.38
RetrievalBEIRRAR-b CodeRAR-b MathMLDRLongEmbed
Base (≤200M)
Best Baseline42.631.847.137.763.2
UltraBERT45.239.155.443.572.0
+ 5 seeds45.25 ± 0.0539.53 ± 0.3054.77 ± 0.6342.84 ± 0.6273.22 ± 0.94
Medium (≤500M)
Best Baseline47.338.654.044.571.7
UltraBERT47.747.061.744.174.9
+ 5 seeds47.73 ± 0.0646.68 ± 0.2760.97 ± 0.5744.00 ± 0.5275.16 ± 0.25

For GLUE, the representative baselines, ModernBERT, Ettin, and NeoBERT, all report the best score of a hyperparameter sweep without repeating over seeds. In contrast, we already use 4 seeds for the small tasks, which fluctuate the most.

As shown in Appendix F.1 and Table 9, the score of Ettin drops from 90.8 to 90.6 when it is fine-tuned with our protocol. We would like to emphasize again that, out of respect for prior work, we adopted existing GLUE and SQuAD 2.0 scores in the literature as our baseline results.


No evaluation on multi-vector ColBERT style IR tasks. The paper states that it follows the ModernBERT evaluation protocol [1] which itself tests on multi-vector IR tasks.

Have the authors tested performance on multi-vector ColBERT style tasks, as [1] does? I'd be curious to see if the advantages of higher masking rates translate over to multi-vector tasks.

Thank you for your suggestion. We fine-tune all baselines and our models in the ColBERT style and evaluate them on the Retrieval tasks of MTEB(eng, v2).

ModelMTEB(eng,v2)
Base (≤200M)
BERT43.9
RoBERTa45.9
MosaicBERT45.9
NomicBERT45.6
GTE-en-MLM46.1
ModernBERT47.4
Ettin Encoder47.6
UltraBERT49.0
Medium (≤500M)
BERT45.2
RoBERTa48.2
GTE-en-MLM45.2
NeoBERT48.4
LFM2.5-Encoder48.0
LFM2.5-Encoder48.8
ModernBERT49.2
Ettin Encoder48.9
UltraBERT50.5

As shown in the table above, the advantage of UltraBERT on single-vector retrieval carries over to ColBERT-style late-interaction retrieval, where it remains the best (49.0 vs 47.6 at the base size and 50.5 vs 49.2 at the medium size).

We have added the experimental details and full results to Appendix C.2 (Table 7) of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 9tag (Part 2/3)

Avey-B tests different masking ratios for a non-transformer encoder and shows (in tables 8 and 9 of Avey-B) that sequence classification peaks at 20% while IR is best at 40-50%, in line with UltraBERT's finding that retrieval favors higher masking ratios than classification. Although this is done at a much smaller token budget, this could serve as useful context. Have the authors considered comparing against Avey-B, which is also shown to achieve strong performance on IR and long context tasks?

Thank you for your suggestion. As you mentioned, Avey-B [1] shows distinct masking ratio preferences, in particular that retrieval prefers higher masking ratios, as in [2] and [3]. We already cite these prior works in the paper, and we have added Avey-B as a citation in the revised manuscript.

Beyond this observation on masking ratio preferences, we find two interesting points in Avey-B:

  • In its masking ratio ablation, Avey-B obtains the best results on SC, TC, and QA at 20–30%, even though 10% is also tested. As the ablation appears to use 10B tokens, our findings suggest that this could be due to the slow convergence of 10% masking under a small budget. With a sufficient budget (such as 1T tokens), lower ratios may be preferred, as in our Section 3.2.
  • Although IR is best at 40–50%, Avey-B finally adopts 20%, since a single masking ratio has to accept this trade-off. Our masking ratio mixing with decoupled LM heads could be applied here as well.

It would therefore be interesting to see whether our strategy and findings can also be extended to non-Transformer encoders.

In addition, we evaluate the publicly released Avey-B models with our benchmark suite. As Avey-B appears to require higher learning rates, we additionally sweep learning rates up to 10× larger for it.

ClassificationGLUESQuAD 2.0MultiCoNER v2Few-NERDRewardBench
Base (≤200M)
BERT84.776.353.866.659.4
Avey-B85.776.953.967.064.5
ModernBERT88.486.757.466.768.0
UltraBERT89.589.257.967.473.0
Medium (≤500M)
BERT85.281.857.767.862.6
Avey-B86.781.357.767.664.6
ModernBERT90.489.860.967.672.8
UltraBERT91.090.963.568.378.8
RetrievalBEIRRAR-b CodeRAR-b MathMLDRLongEmbed
Base (≤200M)
BERT36.76.627.529.029.3
Avey-B41.610.032.736.465.2
ModernBERT42.629.646.137.763.2
UltraBERT45.239.155.443.572.0
Medium (≤500M)
BERT38.612.233.029.830.0
Avey-B44.115.135.835.762.5
ModernBERT45.234.046.139.963.5
UltraBERT47.747.061.744.174.9
  • Avey-B outperforms BERT on all benchmarks at the base size. At the medium size, it also outperforms BERT on all retrieval benchmarks, and matches or outperforms BERT on classification except for two benchmarks.
  • Compared with ModernBERT, Avey-B is ahead on Few-NERD at the base size (67.0 vs 66.7) and equal at the medium size (67.6), and its base model is also ahead on LongEmbed (65.2 vs 63.2). These trends are similar to those reported in the Avey-B paper.
  • UltraBERT achieves the best score on every benchmark in this comparison.
AuthorsOpenReview ↗

Response to Reviewer 9tag (Part 3/3)

Can the authors explain their choice of the subset of benchmarks used in section 3 with dropping SQuAD in tables 1 and 2, and not including NER? Particularly, the difference between the benchmark scores is reported to be much less drastic for NER in tables 8 and 9 of Avey-B with different masking ratios, compared to sequence classification and IR. I am curious how much the observed effects of differences in masking ratio (and of some of the other ablations) could be benchmark dependent.

We kept one representative benchmark for each task family from Section 3.3 on, since SQuAD 2.0 showed a trend similar to GLUE throughout Sections 3.1 and 3.2. For example, we add the SQuAD 2.0 scores to the results of LM head decoupling below:

Masking RatioGLUEBEIRSQuAD 2.0
6%89.543.588.5
15%88.943.887.6
30%88.344.286.7
50%87.545.086.2
Mixing
U(5%,50%)88.044.287.1
Decoupling89.445.188.3

LM head decoupling stays close to 6% masking on both GLUE (89.5 vs 89.4) and SQuAD 2.0 (88.5 vs 88.3). Likewise, U(5%, 50%) (≈ 27.5%) drops clearly on both, similar to 30% masking.

We also evaluate NER across masking ratios:

Masking RatioMultiCoNER v2Few-NERD
3%58.21 ± 0.2067.64 ± 0.05
6%58.62 ± 0.0767.36 ± 0.04
15%58.68 ± 0.1767.30 ± 0.09
30%58.09 ± 0.2167.17 ± 0.03
50%57.07 ± 0.7967.12 ± 0.17

Both NER benchmarks tend to improve as the masking ratio decreases, although the differences are not large. This is roughly in line with Avey-B [1]. With a smaller token budget for Avey-B, the results are more affected by the difference in convergence speed across masking ratios, which could make the trend harder to interpret.


In summary, the revised paper adds the ColBERT results (Appendix C.2, Table 7) and the citation of Avey-B. The changes are marked in blue. We hope these responses address your concerns, and we are happy to answer any further questions.

References
[1] Devang Acharya, , Mohammad Hammoud. "Avey-B." The Fourteenth International Conference on Learning Representations. 2026.
[2] Boizard, Nicolas, et al. "Eurobert: Scaling multilingual encoders for european languages." Second Conference on Language Modeling. 2025.
[3] Gisserot-Boukhlef, Hippolyte, et al. "Should we still pretrain encoders with masked language modeling?." The Fourteenth International Conference on Learning Representations. 2026.

Reviewer_CMv8OpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper revisits the masking recipe for language-model pretraining in the modern regime, where training is scaled up to 8.8 trillion tokens. The authors find that saturation and masking preferences depend on the task: lower masking ratios favor classification, while higher ratios favor retrieval. UltraBERT combines low- and high-mask sequences with separate prediction heads and applies teacher distillation to the low-mask branch. The resulting encoders reduce the trade-off between classification and retrieval, achieving state-of-the-art performance across both task families.

Critical strengths
  1. The paper is very well written and easy to follow. It moves clearly from the pretraining study to the masking experiments and the final UltraBERT recipe.
  2. The different masking preferences across tasks are particularly interesting and motivate a simple, practical pretraining recipe.
  3. I appreciate the scale of the empirical study. The authors explore several model sizes, masking ratios, and very long training runs.
  4. The evaluation includes recent encoder baselines. I find the claims reasonable and supported by the experiments. Cooling down intermediate checkpoints before evaluation also strengthens the saturation analysis.
  5. I appreciate that the authors report negative results, including weaker performance from distillation at high masking ratios. These findings help explain why distillation is applied selectively and provide useful guidance for others designing pretraining recipes.
Critical weaknesses

Major:

  1. The main limitation of the paper is the lack of evaluation on the recent, broader MTEB(v2) benchmark. The existing experiments cover a wide range of tasks, but a broader embedding evaluation would help establish whether the observed masking preferences extend to tasks such as clustering and semantic textual similarity. The reliance on GLUE is also a concern, given criticisms of the benchmark.

Minor:

  1. I suggest clarifying that when saturation appears may depend on the model family, data mixture, masking objective, and downstream adaptation protocol.
  2. Prior work has found that the preferred masking rate can depend on model size. I would be interested to know whether the authors considered adjusting the masking percentages or the ratio of low- to high-mask sequences as model size increases.
  3. A few implementation details would help with reproducibility. In particular, how are the losses normalized and weighted across the 4:1 mixture of 5%- and 50%-masked sequences?
Questions for authors
  1. Did you consider varying the masking percentages or the ratio of low- to high-mask sequences with model size, given prior findings that the preferred masking rate depends on scale?
  2. How are the two branches' losses normalized and weighted?
6 comments
AuthorsOpenReview ↗

Response to Reviewer CMv8 (Part 1/2)

We thank the reviewer for the detailed review, and we are glad that the task-dependent masking preferences and the negative results on distillation were found interesting and useful. We respond to each point below.


(Major) The main limitation of the paper is the lack of evaluation on the recent, broader MTEB(v2) benchmark. The existing experiments cover a wide range of tasks, but a broader embedding evaluation would help establish whether the observed masking preferences extend to tasks such as clustering and semantic textual similarity. The reliance on GLUE is also a concern, given criticisms of the benchmark.

Thank you for the suggestion. As you pointed out, recent embedding models cover various tasks, including not only retrieval but also clustering and STS. In summary, MTEB(eng, v2) shares the same divergence of masking ratio preferences as observed in our paper. Our decoupling strategy accommodates these preferences, resulting in a better average score.

We evaluate the models used for BEIR on MTEB(eng, v2), i.e., the 150M encoders pre-trained at each masking ratio and fine-tuned on MS MARCO (Figure 6 and Table 1), without further fine-tuning. The results are added to Section 3.4 (Figure 10) and Appendix C.3 (Figure 13) of the revised manuscript. We recommend looking at the figures in the revised manuscript, as the trends are easier to see there. We also show the results below:

Masking RatioClassClustPairClsRerankRetrSTSSummAvg
3%68.340.281.144.545.675.728.057.7
6%67.741.681.045.045.476.325.957.9
15%68.542.080.744.345.876.127.458.2
30%69.341.580.744.547.175.624.858.4
50%69.342.580.244.547.674.526.658.5
{5%, 50%} + Decoupling69.242.581.645.347.675.828.058.9
  • Classification, Clustering, and Retrieval: They tend to improve as the masking ratio increases. For Classification, this could seem contradictory to Section 3.2, but MTEB Classification is closer to Retrieval than to fine-tuned classification. With the backbone frozen, a logistic regression model trained on a few samples per label retrieves the best-matching class for each embedding, so the accuracy depends on how well the classes are already separated in the embedding space. The trend indeed reverses with full fine-tuning (see below).
  • Pair Classification and STS: They tend to decrease as the masking ratio increases. Unlike Retrieval, both tasks are scored on the similarity values themselves. The scores need to be precise across all sentence pairs and reflect even subtle differences between two sentences. This takes the same discriminative ability that fine-tuned classification relies on, and the trend is similar to Section 3.2.
  • Reranking and Summarization: We do not observe any clear and distinct trend.
  • {5%, 50%} + Decoupling: It stays at or near the top of every task type. On Retrieval, it matches 50% masking, as on BEIR. However, other task types, including Pair Classification and STS, hold back the average score of 50% masking, so the average plateaus as the masking ratio increases. The decoupled model resolves this conflict and therefore achieves a gain of 0.4 on average.

These results suggest that even among embedding tasks, different task types prefer different masking ratios, and that LM head decoupling accommodates these preferences, making the encoder a more favorable backbone for multi-task benchmarks such as MTEB. Note that the paper examines retrieval on BEIR, and the same trend is reproduced on MTEB, so these results do not contradict the paper but extend it.

In addition, we evaluate the Classification tasks of MTEB(eng, v2) with full fine-tuning, using the hyperparameter sweep of GLUE. As in Section 3.2, the accuracy decreases as the masking ratio increases. As mentioned above, this indicates that the opposite trend on MTEB comes from its logistic regression on frozen embeddings.

Masking RatioMTEB(eng, v2) Class.
3%93.1
6%93.09
15%92.95
30%92.82
50%92.83

Regarding GLUE, we are aware of the concerns about the benchmark. However, it is still the main classification benchmark for recent encoders such as ModernBERT and Ettin, and NeoBERT explores its architectural modifications directly on GLUE. GLUE was also more challenging in our experiments, showing a gap of 2.3 between the best and worst masking ratios. In contrast, the encoders fine-tuned on MTEB Classification already reach about 93% accuracy and differ by only 0.28. Still, we do not rely on GLUE alone. Section 3.2 also reports SQuAD 2.0, and Table 3 covers 4 more classification benchmarks.

AuthorsOpenReview ↗

Response to Reviewer CMv8 (Part 2/2)

I suggest clarifying that when saturation appears may depend on the model family, data mixture, masking objective, and downstream adaptation protocol.

Thank you for the suggestion. We now state the scope of the saturation claim in Section 3.1 of the revised paper. All of our observations are under the MLM objective with a single architecture family and data mixture, so the saturation point may differ in other settings, as you suggested.

Within this scope, we observe saturation at two model sizes (150M and 1.3B), on GLUE, SQuAD 2.0, and BEIR with fine-tuning, and on 6 MCQA benchmarks with instruction tuning. We also find that its onset depends on the masking ratio, around 1–2T tokens at 15% and around 2–3T tokens at 6% and 3% (footnote 1).


Prior work has found that the preferred masking rate can depend on model size. I would be interested to know whether the authors considered adjusting the masking percentages or the ratio of low- to high-mask sequences as model size increases.

Did you consider varying the masking percentages or the ratio of low- to high-mask sequences with model size, given prior findings that the preferred masking rate depends on scale?

As discussed in Sections 3.2 and 3.3, we find that across all model sizes, (1) classification favors lower masking ratios and (2) retrieval favors higher ones. Larger models need more tokens to converge, but the token budgets of prior work were fixed and insufficient for convergence. Since higher masking ratios converge faster, the optimal ratio appears to shift upward for larger models.

In more detail, [1] and [2] report that 20% is optimal for the base model (124M) and 40% for the large model (354M) on GLUE, and [3] reports that 40% is optimal for the 1B model on sequence classification. In Section 3.2 and Figure 4, we cover all three sizes (150M, 380M, and 1.3B) with a wide range of masking ratios that includes their optimal ones. Consistently, ratios lower than their optimal ones perform better.

Therefore, we conclude that the masking ratio preference is consistent regardless of the model size once the encoders are trained to saturation, and we designed the training method of UltraBERT accordingly.


A few implementation details would help with reproducibility. In particular, how are the losses normalized and weighted across the 4:1 mixture of 5%- and 50%-masked sequences?

How are the two branches' losses normalized and weighted?

Each branch's loss is averaged over its own masked tokens across the entire global batch, and the two are combined as $ 0.8\mathcal{L}_{5\%} + 0.2\mathcal{L}_{50\%}, $ where \mathcal{L}_{5\\%} and \mathcal{L}_{50\\%} denote the losses on the 5%- and 50%-masked sequences, respectively.

We have added this to Appendix A.1 of the revised paper.


In summary, the revised paper adds the MTEB(eng, v2) evaluation (Section 3.4 and Appendix C.3), states the scope of the saturation claim (Section 3.1), and describes the loss weighting of the two masking ratios (Appendix A.1). The changes are marked in blue. We hope these responses address your concerns, and we are happy to answer any further questions.

References
[1] Wettig, Alexander, et al. "Should you mask 15% in masked language modeling?." Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2023.
[2] Dong, Mingze, Leda Wang, and Yuval Kluger. "Understanding and enhancing mask-based pretraining towards universal representations." Advances in Neural Information Processing Systems 38 (2026): 142696-142739.
[3] Gisserot-Boukhlef, Hippolyte, et al. "Should we still pretrain encoders with masked language modeling?." International Conference on Learning Representations. Vol. 2026. 2026.

AuthorsOpenReview ↗

Gentle Reminder to Reviewer CMv8

Thank you again for your time and effort in reviewing our paper. As the discussion period is coming to an end, we would like to kindly ask whether our responses and the revised paper have addressed your concerns. If there are any remaining questions, we would be happy to answer them.

Reviewer_bSk7OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

UltraBERT revisits masked language model pretraining in the mordern era and training to the trillion token scale. The train models at the 150M, 380M and 1.3B parameter scale and up to 8.8T tokens. The main findings from the paper are as follows: 1) On downstream performance (GLUE, SQUAD etc), the performance saturates at around a quarter of the training budget while the val loss keeps falling. 2)Lower masking rations are better for classification and higher ratios are better for retrieval once we are at saturation 3) Logit distillation from a 1.3B teacher helps classification but hurts retrieval at 50% masked rate. In general this work is reviewing design choices that are usually taken for granted for BERT style models at larger param and token scales.

Critical strengths
  1. Thorough emperical investigation of encoder pretraining at substantial scale: The scaling study i.e scaling the models to 1.3B and also the tokens to 8.8T are very useful and the findings would be very useful for the community.

  2. The results and insights on how mixing masking ratios allows to get good resultson both classification and retrieval is very insightful! Also good insights on the paper on things like saturation with trianing tokens etc.

  3. The downstream evaluation is actually comprehensive and covers multiple tasks and the results on them are great! Its wonderful to see that UltraBERT performs well on both classification and retrieval tasks.

Critical weaknesses
  1. Potential evaluation bias: In Appendix B.2 the paper indicates that the hyperparameters for retrieval and the checkpoints are selected using BEIR scores. This could bias the reported results. Can the authors clarify this ?

  2. Insufficient final-model ablations: The final model combines head decoupling, masking-ratio mixing, pruning, and selective distillation without fully isolating their contributions under the final training configuration.

  3. The claim that encoder performance saturates beyond 1–2T tokens is based on a small set of architectures, masking ratios, and downstream tasks, I still think the point stands but the author's should consider toning the claim.

Questions for authors
  1. What if the masking ratio is just a range e.g 15% to 50% and not one of both, does that work better ?

  2. Does the reported saturation happen across different masking ratios/pre-training obectives or is it specific to your configuration.

5 comments
AuthorsOpenReview ↗

Response to Reviewer bSk7 (Part 1/2)

We thank the reviewer for the careful review, and we are glad that the scaling study and the mixing of masking ratios were found useful and insightful. We respond to each point below.


Potential evaluation bias: In Appendix B.2 the paper indicates that the hyperparameters for retrieval and the checkpoints are selected using BEIR scores. This could bias the reported results. Can the authors clarify this ?

We agree that selecting checkpoints on BEIR could bias the BEIR scores. As mentioned in Appendix B.2, we report the highest BEIR score to measure the best attainable retrieval performance of each backbone. However, when we instead select checkpoints on held-out datasets, UltraBERT remains the best at both sizes, as on BEIR.

Specifically, we choose the checkpoints on held-out NanoBEIR datasets and evaluate on the Retrieval tasks of MTEB(eng, v2). In detail, we use 5 datasets in NanoBEIR: NanoDBPedia, NanoNFCorpus, NanoNQ, NanoQuoraRetrieval, and NanoSciFact. The 7 datasets that overlap with MTEB(eng, v2) are excluded, and the Nano version of MS MARCO is also excluded as MS MARCO is used for fine-tuning.

ModelNanoBEIRMTEB (eng,v2)
Base (≤200M)
BERT52.4038.32
RoBERTa54.3841.82
MosaicBERT58.0143.10
NomicBERT58.9943.40
GTE-en-MLM59.2743.71
ModernBERT57.3644.70
Ettin Encoder58.6745.71
UltraBERT61.4047.41
Medium (≤500M)
BERT56.4341.39
RoBERTa59.3144.71
GTE-en-MLM60.1146.54
NeoBERT59.4044.03
LFM2.5-Encoder (230M)63.1147.68
LFM2.5-Encoder (355M)63.8049.48
ModernBERT60.2647.11
Ettin Encoder62.4647.98
UltraBERT64.4450.24

The results are consistent with the BEIR results: UltraBERT outperforms the second best by 1.7 at the base size (47.4 vs 45.7) and by 0.7 at the medium size (50.2 vs 49.5). For BEIR in Table 3, the margins of UltraBERT are 2.6 and 0.4, respectively. In addition, we observe that the learning rates selected for UltraBERT are the same as those selected on BEIR: 5e-5 for base and 3e-5 for medium.

We have added the full results to Appendix C.1 (Table 6) of the revised paper.


Insufficient final-model ablations: The final model combines head decoupling, masking-ratio mixing, pruning, and selective distillation without fully isolating their contributions under the final training configuration.

While we have not ablated each component of the final model, we note that our study is explorative rather than a search over design choices: each component was settled by its own controlled comparison, hence the final model becoming the composition of those outcomes.

(1) We tracked intermediate checkpoints of 150M and 1.3B encoders trained on up to 8.8T tokens, locating the saturation point that sets our token budget (Section 3.1). (2) We compared single ratios, decay, uniform sampling, and {5%,50%} with and without decoupling; only decoupled heads recover both tasks, consistent with the LM head geometries diverging between the two regimes (Section 3.2–3.4). (3) We compared distilled and from-scratch 380M models at 15% and 50% masking, finding the gains confined to the low ratio (Section 3.5).

Each comparison isolates one component under matched conditions, and none of the components was introduced to compensate for another. We therefore expect them to combine without interfering with each other, and the findings to remain the same at the final scale. Although an ablation there would be more complete, we judged it to add little beyond what these comparisons establish for the pre-training cost it requires.

AuthorsOpenReview ↗

Response to Reviewer bSk7 (Part 2/2)

The claim that encoder performance saturates beyond 1–2T tokens is based on a small set of architectures, masking ratios, and downstream tasks, I still think the point stands but the author's should consider toning the claim.

Does the reported saturation happen across different masking ratios/pre-training obectives or is it specific to your configuration.

Thank you for the suggestion. We answer the two points together, since both concern the scope of the saturation claim.

We agree that the scope should be stated explicitly, and we now state it in Section 3.1 of the revised paper. All of our observations are under the MLM objective, with a single architecture family and data mixture, and we did not examine token scaling under other pre-training objectives such as RTD.

Within this scope, we observed saturation in several settings. For model sizes, both the 150M and 1.3B encoders saturate (Figure 2). For downstream tasks, it appears on GLUE, SQuAD 2.0, and BEIR with fine-tuning (Figure 2), and on 6 MCQA benchmarks with instruction tuning (Figure 3). For masking ratios, Section 3.1 uses 30%, and we also located the saturation point at the other ratios, which is how the token budgets in Section 3.2 were chosen (footnote 1). Saturation occurs around 1–2T tokens at 15% and around 2–3T tokens at 6% and 3%, and Figure 6(c, d) shows the curves at 6% up to 5T tokens. Therefore, saturation is not specific to one masking ratio: it appears at every ratio from 3% to 50%, and only its onset shifts, with lower ratios saturating later.


What if the masking ratio is just a range e.g 15% to 50% and not one of both, does that work better ?

We would like to point out that U(5\mathcal{U}(5\\%,50\\%) in Table 1 denotes sampling the masking ratio uniformly from 5% to 50%, a strategy proposed by [1], which we include as a baseline.

In short, sampling from a range behaves like its average masking ratio. The average ratio of U(5\mathcal{U}(5\\%,50\\%) is 27.5%, and its results are close to those of the nearest single ratio, 30% (88.0 vs 88.3 on GLUE, and the same 44.2 on BEIR). We also observe that its LM head geometry is similar to that of 30% in Figure 9(a) and (b).

This is why we propose two distinct masking ratios with LM head decoupling, as using continuous masking ratios with a shared LM head falls back to an intermediate masking ratio and its trade-off.


In summary, the revised paper adds the retrieval evaluation with held-out checkpoint selection (Appendix C.1, Table 6) and states the scope of the saturation claim (Section 3.1). The changes are marked in blue. We hope these responses address your concerns, and we are happy to answer any further questions.

References
[1] Dong, Mingze, Leda Wang, and Yuval Kluger. "Understanding and enhancing mask-based pretraining towards universal representations." Advances in Neural Information Processing Systems 38 (2026): 142696-142739.

AuthorsOpenReview ↗

Gentle Reminder to Reviewer bSk7

Thank you again for your time and effort in reviewing our paper. As the discussion period is coming to an end, we would like to kindly ask whether our responses and the revised paper have addressed your concerns. If there are any remaining questions, we would be happy to answer them.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the Reviews and Responses

We sincerely thank all reviewers for their thoughtful and constructive reviews. We are encouraged that the reviewers found the paper well written and easy to follow (@CMv8, @9tag), and our large-scale study of pre-training saturation useful and insightful (@bSk7, @CMv8, @9tag). The reviewers also considered LM head decoupling an insightful and practical way to achieve both classification and retrieval (@bSk7, @CMv8, @9tag), and the evaluation comprehensive with strong results (@bSk7, @9tag).

We summarize below how we addressed the reviewers' concerns. All additional results are consistent with the findings of the paper.

  • Checkpoint selection on BEIR (@bSk7): We instead select checkpoints on held-out NanoBEIR datasets and evaluate on the Retrieval tasks of MTEB(eng, v2). UltraBERT remains the best at both sizes, with 47.4 vs 45.7 at the base size and 50.2 vs 49.5 at the medium size (Appendix C.1, Table 6).
  • Multi-vector retrieval (@9tag): We fine-tune all baselines and our models in the ColBERT style and evaluate them on the same Retrieval tasks. UltraBERT again remains the best at both sizes, with 49.0 vs 47.6 at the base size and 50.5 vs 49.2 at the medium size (Appendix C.2, Table 7).
  • Masking ratio preferences on MTEB(eng, v2) (@CMv8): We evaluate the encoders pre-trained at each masking ratio on all task types of MTEB(eng, v2). It shares the same divergence of masking ratio preferences as observed in our paper, and LM head decoupling achieves the best average score (Section 3.4, Figure 10, and Appendix C.3).
  • Comparison with Avey-B (@9tag): We discuss the masking ratio preferences observed in Avey-B in relation to our findings, and evaluate the publicly released Avey-B models with our benchmark suite. UltraBERT achieves the best score on every benchmark in this comparison.
  • Choice of benchmarks in Section 3 (@9tag): We provide the SQuAD 2.0 scores for the results of LM head decoupling, which show a trend similar to GLUE. We also evaluate NER across masking ratios, and both NER benchmarks tend to improve as the masking ratio decreases.
  • Seed variance (@9tag): We re-evaluate all main benchmarks with 5 seeds for both UltraBERT models. The superiority of UltraBERT remains unchanged.

Following the reviewers' suggestions, we also clarify the setting of our saturation analysis (@bSk7, @CMv8), and add the loss weighting details (@CMv8) and the citation of Avey-B (@9tag). The changes in the revised paper are marked in blue. We thank the reviewers again for their suggestions, which helped us improve the paper.

AuthorsOpenReview ↗

Closing Remarks from the Authors

Dear Reviewers and Chairs,

Thank you for your time and effort. We truly appreciate that you have helped us and our research in so many ways. We are also grateful that we were given ample opportunity to explain our work.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.