- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This work studies the effects of pre-training recipes for encoder models at scale, and uses the findings to train a new encoder model (UltraBERT) that is shown to achieve SOTA performance on a wide range of classification and information retrieval benchmarks at both a base and medium model size. It finds that downstream performance of encoder models saturates with a large number of pre-training tokens unlike decoder models, that there's a trade-off in the masking ratio for performance on classification and retrieval tasks, and shows that pre-training using LM head decoupling can achieve the best of both worlds.
Critical strengths
- The paper is well written and easy to follow.
- The multiple studies on the effect of large scale training on encoders are insightful, particularly the pre-training saturation and the link between masking ratio and the effective rank of the representations.
- Using decoupled LM heads to achieve the best of both worlds is insightful.
- UltraBERT achieves strong performance on downstream benchmarks.
Critical weaknesses
- Only 4 seeds are used for the small GLUE tasks, 3 for MultiCoNER (for selection), and the rest of the seed counts do not seem to be reported. Seed variances and statistics are also not reported. One of the headline results is 91.0 on GLUE with the second best model obtaining 90.8; it's not clear how far ahead UltraBERT is, without the seed variance being shown.
- No evaluation on multi-vector ColBERT style IR tasks. The paper states that it follows the ModernBERT evaluation protocol [1] which itself tests on multi-vector IR tasks.
[1] Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Griffin Thomas Adams, Jeremy Howard, and Iacopo Poli. Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025.
Questions for authors
- [2] tests different masking ratios for a non-transformer encoder and shows (in tables 8 and 9 of [2]) that sequence classification peaks at 20% while IR is best at 40-50%, in line with UltraBERT's finding that retrieval favors higher masking ratios than classification. Although this is done at a much smaller token budget, this could serve as useful context. Have the authors considered comparing against [2], which is also shown to achieve strong performance on IR and long context tasks?
- Could the authors report seed SD and possibly run evals with more seeds (at least 5-10)?
- Have the authors tested performance on multi-vector ColBERT style tasks, as [1] does? I'd be curious to see if the advantages of higher masking rates translate over to multi-vector tasks.
- Can the authors explain their choice of the subset of benchmarks used in section 3 with dropping SQuAD in tables 1 and 2, and not including NER? Particularly, the difference between the benchmark scores is reported to be much less drastic for NER in tables 8 and 9 of [2] with different masking ratios, compared to sequence classification and IR. I am curious how much the observed effects of differences in masking ratio (and of some of the other ablations) could be benchmark dependent.
[2] Devang Acharya and Mohammad Hammoud. Avey-B. In The Fourteenth International Conference on Learning Representations, 2026.
5 comments
Response to Reviewer 9tag (Part 2/3)
Avey-B tests different masking ratios for a non-transformer encoder and shows (in tables 8 and 9 of Avey-B) that sequence classification peaks at 20% while IR is best at 40-50%, in line with UltraBERT's finding that retrieval favors higher masking ratios than classification. Although this is done at a much smaller token budget, this could serve as useful context. Have the authors considered comparing against Avey-B, which is also shown to achieve strong performance on IR and long context tasks?
Thank you for your suggestion. As you mentioned, Avey-B [1] shows distinct masking ratio preferences, in particular that retrieval prefers higher masking ratios, as in [2] and [3]. We already cite these prior works in the paper, and we have added Avey-B as a citation in the revised manuscript.
Beyond this observation on masking ratio preferences, we find two interesting points in Avey-B:
- In its masking ratio ablation, Avey-B obtains the best results on SC, TC, and QA at 20–30%, even though 10% is also tested. As the ablation appears to use 10B tokens, our findings suggest that this could be due to the slow convergence of 10% masking under a small budget. With a sufficient budget (such as 1T tokens), lower ratios may be preferred, as in our Section 3.2.
- Although IR is best at 40–50%, Avey-B finally adopts 20%, since a single masking ratio has to accept this trade-off. Our masking ratio mixing with decoupled LM heads could be applied here as well.
It would therefore be interesting to see whether our strategy and findings can also be extended to non-Transformer encoders.
In addition, we evaluate the publicly released Avey-B models with our benchmark suite. As Avey-B appears to require higher learning rates, we additionally sweep learning rates up to 10× larger for it.
| Classification | GLUE | SQuAD 2.0 | MultiCoNER v2 | Few-NERD | RewardBench |
|---|---|---|---|---|---|
| Base (≤200M) | |||||
| BERT | 84.7 | 76.3 | 53.8 | 66.6 | 59.4 |
| Avey-B | 85.7 | 76.9 | 53.9 | 67.0 | 64.5 |
| ModernBERT | 88.4 | 86.7 | 57.4 | 66.7 | 68.0 |
| UltraBERT | 89.5 | 89.2 | 57.9 | 67.4 | 73.0 |
| Medium (≤500M) | |||||
| BERT | 85.2 | 81.8 | 57.7 | 67.8 | 62.6 |
| Avey-B | 86.7 | 81.3 | 57.7 | 67.6 | 64.6 |
| ModernBERT | 90.4 | 89.8 | 60.9 | 67.6 | 72.8 |
| UltraBERT | 91.0 | 90.9 | 63.5 | 68.3 | 78.8 |
| Retrieval | BEIR | RAR-b Code | RAR-b Math | MLDR | LongEmbed |
|---|---|---|---|---|---|
| Base (≤200M) | |||||
| BERT | 36.7 | 6.6 | 27.5 | 29.0 | 29.3 |
| Avey-B | 41.6 | 10.0 | 32.7 | 36.4 | 65.2 |
| ModernBERT | 42.6 | 29.6 | 46.1 | 37.7 | 63.2 |
| UltraBERT | 45.2 | 39.1 | 55.4 | 43.5 | 72.0 |
| Medium (≤500M) | |||||
| BERT | 38.6 | 12.2 | 33.0 | 29.8 | 30.0 |
| Avey-B | 44.1 | 15.1 | 35.8 | 35.7 | 62.5 |
| ModernBERT | 45.2 | 34.0 | 46.1 | 39.9 | 63.5 |
| UltraBERT | 47.7 | 47.0 | 61.7 | 44.1 | 74.9 |
- Avey-B outperforms BERT on all benchmarks at the base size. At the medium size, it also outperforms BERT on all retrieval benchmarks, and matches or outperforms BERT on classification except for two benchmarks.
- Compared with ModernBERT, Avey-B is ahead on Few-NERD at the base size (67.0 vs 66.7) and equal at the medium size (67.6), and its base model is also ahead on LongEmbed (65.2 vs 63.2). These trends are similar to those reported in the Avey-B paper.
- UltraBERT achieves the best score on every benchmark in this comparison.
Response to Reviewer 9tag (Part 3/3)
Can the authors explain their choice of the subset of benchmarks used in section 3 with dropping SQuAD in tables 1 and 2, and not including NER? Particularly, the difference between the benchmark scores is reported to be much less drastic for NER in tables 8 and 9 of Avey-B with different masking ratios, compared to sequence classification and IR. I am curious how much the observed effects of differences in masking ratio (and of some of the other ablations) could be benchmark dependent.
We kept one representative benchmark for each task family from Section 3.3 on, since SQuAD 2.0 showed a trend similar to GLUE throughout Sections 3.1 and 3.2. For example, we add the SQuAD 2.0 scores to the results of LM head decoupling below:
| Masking Ratio | GLUE | BEIR | SQuAD 2.0 |
|---|---|---|---|
| 6% | 89.5 | 43.5 | 88.5 |
| 15% | 88.9 | 43.8 | 87.6 |
| 30% | 88.3 | 44.2 | 86.7 |
| 50% | 87.5 | 45.0 | 86.2 |
| Mixing | |||
| U(5%,50%) | 88.0 | 44.2 | 87.1 |
| Decoupling | 89.4 | 45.1 | 88.3 |
LM head decoupling stays close to 6% masking on both GLUE (89.5 vs 89.4) and SQuAD 2.0 (88.5 vs 88.3). Likewise, U(5%, 50%) (≈ 27.5%) drops clearly on both, similar to 30% masking.
We also evaluate NER across masking ratios:
| Masking Ratio | MultiCoNER v2 | Few-NERD |
|---|---|---|
| 3% | 58.21 ± 0.20 | 67.64 ± 0.05 |
| 6% | 58.62 ± 0.07 | 67.36 ± 0.04 |
| 15% | 58.68 ± 0.17 | 67.30 ± 0.09 |
| 30% | 58.09 ± 0.21 | 67.17 ± 0.03 |
| 50% | 57.07 ± 0.79 | 67.12 ± 0.17 |
Both NER benchmarks tend to improve as the masking ratio decreases, although the differences are not large. This is roughly in line with Avey-B [1]. With a smaller token budget for Avey-B, the results are more affected by the difference in convergence speed across masking ratios, which could make the trend harder to interpret.
In summary, the revised paper adds the ColBERT results (Appendix C.2, Table 7) and the citation of Avey-B. The changes are marked in blue. We hope these responses address your concerns, and we are happy to answer any further questions.
References
[1] Devang Acharya, , Mohammad Hammoud. "Avey-B." The Fourteenth International Conference on Learning Representations. 2026.
[2] Boizard, Nicolas, et al. "Eurobert: Scaling multilingual encoders for european languages." Second Conference on Language Modeling. 2025.
[3] Gisserot-Boukhlef, Hippolyte, et al. "Should we still pretrain encoders with masked language modeling?." The Fourteenth International Conference on Learning Representations. 2026.
Response to Reviewer 9tag (Part 1/3)
We thank the reviewer for the thorough review, and we are glad that the paper was found well written, and the studies on pre-training saturation and the decoupled LM heads insightful. We respond to each point below.
We re-evaluate all main benchmarks with 5 seeds for our two models (UltraBERT-150M and UltraBERT-380M). We confirm that the superiority of UltraBERT remains unchanged under multi-seed evaluation.
For classification, we repeat the evaluation protocol 5 times with different seeds. For retrieval, we repeat the MS MARCO fine-tuning 5 times with different seeds and evaluate each model, and the MLDR fine-tuning continues from each of these MS MARCO models.
For GLUE, the representative baselines, ModernBERT, Ettin, and NeoBERT, all report the best score of a hyperparameter sweep without repeating over seeds. In contrast, we already use 4 seeds for the small tasks, which fluctuate the most.
As shown in Appendix F.1 and Table 9, the score of Ettin drops from 90.8 to 90.6 when it is fine-tuned with our protocol. We would like to emphasize again that, out of respect for prior work, we adopted existing GLUE and SQuAD 2.0 scores in the literature as our baseline results.
Thank you for your suggestion. We fine-tune all baselines and our models in the ColBERT style and evaluate them on the Retrieval tasks of MTEB(eng, v2).
As shown in the table above, the advantage of UltraBERT on single-vector retrieval carries over to ColBERT-style late-interaction retrieval, where it remains the best (49.0 vs 47.6 at the base size and 50.5 vs 49.2 at the medium size).
We have added the experimental details and full results to Appendix C.2 (Table 7) of the revised manuscript.