- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies continual pretraining under a fixed token budget as a joint acquisition and retention problem. It introduces Capability Budgeting, a framework that allocates training tokens across six data pools serving different roles and evaluates trained candidates against explicit per-capability loss budgets on nine protected tasks. Using Qwen3-4B-Base and Qwen3-8B-Base, three allocations and three seeds at 16.384M tokens per run yield 18 checkpoints. All of them improve target-language average mathematics on MGSM by 2.27 to 4.27 points, full-test English GSM8K, and Swahili Global-MMLU.
Critical strengths
- The paper frames data allocation and capability retention as one explicit selection criterion, which is a clear and reusable way to compare adaptation recipes.
- The Bengali result, where mathematics improves while Global-MMLU Bengali drops throughout the matrix, is a useful demonstration that mathematical gain is not a proxy for knowledge retention in the same language.
Critical weaknesses
- Benchmark contamination is audited only for HumanEval. The primary outcome MGSM is never audited against the synthetic mathematical reasoning data that dominates the target-language pools, although such overlap is the most plausible shortcut for the observed gains.
- Budget feasibility is decided on raw point estimates, while the paper's own interval analysis tells a different story: Table 20 shows that under one-sided paired checks most checkpoints fail or remain unknown on the broad tasks. With evaluation subsets of 250 items and margins of 4 points, many pass and fail calls are within sampling noise, so the headline claim that qualifying candidates exist is not statistically robust.
Questions for authors
- Which allocation choices were fixed before the exploratory 8B results were observed, and which were set afterwards? Appendix C suggests the final allocations descend from an exploratory phase.
- Given that omitting the 15-item CMMLU mathematics subset changes no reported quantity, would excluding it from the primary budgets simplify the framework?