acceptodds
Under review as a conference paper at ICLR 2027

Capability Budgeting: Acquiring Mathematical Reasoning While Retaining Capabilities in Multilingual Continual Pretraining

Abstract

Multilingual continual pretraining should acquire target skills while retaining capabilities that remain useful. We study this objective under a fixed token budget through Capability Budgeting: allocating data by role and assessing mathematical acquisition against explicit per-capability loss budgets. A controlled study jointly adapts Thai, Bengali, and Swahili on Qwen3-4B-Base and Qwen3-8B-Base, comparing three allocations with three training seeds each and 16.384M tokens per run. All 18 trained models improve target-language average mathematics accuracy by 2.27–4.27 percentage points, full-test English GSM8K, and Swahili Global-MMLU. Mean GSM8K gains range from 2.70 to 3.92 points at 4B and 1.47 to 1.84 at 8B; all six code-evaluated 4B models gain 6.10–7.93 points on full-test HumanEval. Every matched-seed comparison at both sizes contains a trained candidate combining mathematical gains with all nine measured-score retention requirements. At a fixed combined English-pool share, exchanging mathematical for general text changes the distribution of target-language and broader task outcomes. These differences matter for selection: retention requirements change the highest-mathematics choice in every 4B seed comparison. The results demonstrate mathematical acquisition with budgeted capability retention, and show how data roles and individual task outcomes inform the choice of an adaptation recipe.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multilingual continual pretraining, capability retention, data allocation, mathematical reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_AbGtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies continual pretraining under a fixed token budget as a joint acquisition and retention problem. It introduces Capability Budgeting, a framework that allocates training tokens across six data pools serving different roles and evaluates trained candidates against explicit per-capability loss budgets on nine protected tasks. Using Qwen3-4B-Base and Qwen3-8B-Base, three allocations and three seeds at 16.384M tokens per run yield 18 checkpoints. All of them improve target-language average mathematics on MGSM by 2.27 to 4.27 points, full-test English GSM8K, and Swahili Global-MMLU.

Critical strengths
  • The paper frames data allocation and capability retention as one explicit selection criterion, which is a clear and reusable way to compare adaptation recipes.
  • The Bengali result, where mathematics improves while Global-MMLU Bengali drops throughout the matrix, is a useful demonstration that mathematical gain is not a proxy for knowledge retention in the same language.
Critical weaknesses
  • Benchmark contamination is audited only for HumanEval. The primary outcome MGSM is never audited against the synthetic mathematical reasoning data that dominates the target-language pools, although such overlap is the most plausible shortcut for the observed gains.
  • Budget feasibility is decided on raw point estimates, while the paper's own interval analysis tells a different story: Table 20 shows that under one-sided paired checks most checkpoints fail or remain unknown on the broad tasks. With evaluation subsets of 250 items and margins of 4 points, many pass and fail calls are within sampling noise, so the headline claim that qualifying candidates exist is not statistically robust.
Questions for authors
  • Which allocation choices were fixed before the exploratory 8B results were observed, and which were set afterwards? Appendix C suggests the final allocations descend from an exploratory phase.
  • Given that omitting the 15-item CMMLU mathematics subset changes no reported quantity, would excluding it from the primary budgets simplify the framework?
Reviewer_tBAGOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Capability Budgeting, a framework for improving multilingual mathematical reasoning while limiting measured losses on other capabilities. It allocates a fixed training budget across language-serving, general-English, and mathematical-English data, then selects the highest-performing candidate that satisfies task-specific retention limits.

The study evaluates three data mixtures on Qwen3-4B-Base and Qwen3-8B-Base, with three training seeds and 16.384 million tokens per run. All models improve average Thai, Bengali, and Swahili mathematics by 2.27–4.27 percentage points and improve English GSM8K. Every model-size/seed comparison contains a candidate satisfying all nine measured retention limits.

Critical strengths

All trained models improve average Thai, Bengali, and Swahili maths accuracy by 2.27–4.27 percentage points. All also improve English GSM8K.

Critical weaknesses
  1. Unclear advance over existing approaches. Earlier work already studies multilingual adaptation and data-mixture tradeoffs. This paper tests three preset mixtures without comparing against closely related methods under the same training budget. The results therefore leave unclear how much the proposed framework adds beyond conventional training and model selection.
  2. One accepted model loses 3.2 points on Bengali general knowledge, within the four-point limit. However, its 95% confidence interval extends to a 6.4-point loss. Although the authors acknowledge this uncertainty, the evidence is insufficient to conclude that the model would remain within the limit on unseen questions.
Questions for authors
  1. What does Capability Budgeting add beyond applying per-task loss limits to models trained using existing multilingual adaptation methods?
  2. Would any selected model satisfy all retention limits when accounting for statistical uncertainty? Can you establish this using the existing predictions, completing missing confidence intervals and grouping translations of the same MGSM question together?
Reviewer_ycNbOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies multilingual continual pretraining under a fixed per-run token budget, aiming to improve Thai, Bengali, and Swahili mathematics while limiting losses on individual protected capabilities. Capability Budgeting allocates tokens across six data pools, evaluates trained candidates against task-specific loss thresholds, and selects the feasible candidate with the highest target-language mathematical gain. The experiments compare three allocations on Qwen3-4B-Base and Qwen3-8B-Base using three optimization seeds, yielding 18 checkpoints at 16.384M tokens each. All checkpoints improve target-average MGSM and full-test GSM8K; every matched-seed group contains a candidate satisfying all nine reference constraints. A controlled exchange between dedicated general-English and mathematical-English pools produces task-specific changes, and the retention constraints alter the selected candidate in every 4B seed comparison.

Critical strengths

A concrete demonstration of why capability-specific constraints matter. Bengali mathematical improvement coexists with Bengali Global-MMLU deterioration. In the 4B seed-42 comparison, different protected tasks exclude Replay and English-role, while Uniform remains feasible (Section 4.2). This gives the framework a practical motivation and shows how task-specific requirements can change a deployment choice that target-average accuracy alone would suggest.

Repeated acquisition gains with meaningful evaluation coverage. All 18 checkpoints improve target-average MGSM by 2.27–4.27 percentage points and improve full-test GSM8K. All six code-evaluated 4B checkpoints improve full-test HumanEval by 6.10–7.93 points, with positive reported paired-interval lower bounds (Tables 2, 3, 5, and 18). The matched training setup and complete seed reporting make these descriptive findings credible and provide evidence in favor of the paper.

Critical weaknesses

The incremental contribution is insufficiently established. Equation 2 applies explicit task thresholds to a finite set of evaluated checkpoints, and Equation 3 re-expresses feasibility through normalized losses. The paper demonstrates the usefulness of these decision tools, but the additional knowledge established beyond familiar acquisition–retention trade-offs remains limited. Its central empirical intervention is one exchange between two English pools, tested within three nearby manual allocations, one model family, and one training budget. An empirical contribution can justify acceptance, but the comparison with the closest multilingual CPT studies does not yet make the distinctive, consequential conclusion sufficiently clear (Sections 2, 4.3, and 5). This limits my assessment of the paper's significance.

The stability of the decision-relevant findings is not sufficiently established. Table 22 provides point differences for the key task-specific role-exchange effects without paired uncertainty intervals. The 8B GPQA gains of 2.53, 0.51, and 0.51 points correspond to five, one, and one additional correct answers. Some selection decisions are similarly close to a boundary: Replay at 4B seed 42 exceeds the Bengali knowledge-loss budget by one answer on a 250-item set. Table 20 does not establish a complete pass of its eight-broad-task confidence-bound checks, which are separate from the nine measured-score constraints. I accept the paper's explicit point-score definition; failure to establish a confidence-bound pass does not prove a violation of the underlying retention requirements. However, robustness of the central contrasts and selected candidates remains uncertain. This limits the evidence for the framework as a dependable selection procedure.

Questions for authors

Clarification of contribution: Relative to the closest multilingual CPT studies in Section 5, what is the most important new empirical conclusion, and which specific comparison establishes it? This would materially affect my assessment of informativeness and significance.

Analysis using existing predictions: Can you provide paired intervals for the key task-specific contrasts in Table 22? For aggregate MGSM analyses, can translations be resampled jointly by underlying question, addressing the dependence acknowledged in Appendix B? Please identify any unavailable item-level results.

Clarification and analysis of selection reliability: How stable are the feasible sets and selected candidates in Table 4 under paired evaluation-item resampling? Building on Appendices C–D, please clarify which allocation and threshold decisions preceded prospective results and which were informed by earlier observations. These answers would help assess the reliability of the demonstrated selection procedure.

Reviewer_zydtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Capability Budgeting, a practical framework for multilingual continual pretraining that jointly targets skill acquisition and retention of existing capabilities under explicit per-task loss budgets. Extensive experimental results establish fixed-budget mathematical acquisition with budgeted capability retention and show why the role of training data and the requirements on individual capabilities jointly matter for choosing an adaptation recipe.

Critical strengths
  1. The paper addresses an important and practically relevant problem.

  2. The overall formulation is technically coherent.

  3. The paper is well organized and easy to follow.

Critical weaknesses
  1. The novelty of the proposed method appears limited. The framework is a selection protocol over a small set of fixed allocations rather than an optimization or learning method that finds allocations satisfying budgets automatically.

  2. The method relies heavily on heuristic choices whose effects are not systematically analyzed. The per-task loss budgets are researcher-chosen constants. There is no systematic guidance for setting, calibrating, or validating them against user tolerances or statistical uncertainty.

  3. The experimental evaluation is incomplete. The training budget is relatively small (16.384M tokens, 250 steps), and coverage is limited to a single model family.

  4. The ablation study does not isolate the contributions of all major components. The pool-internal compositions (e.g., components inside each language-serving pool) are not ablated, limiting causal resolution.

  5. No source code is provided, making the results difficult to reproduce. While data provenance is described, the lack of released processed corpora or reconstruction manifests may hinder reproducibility of the exact mixtures and screening.

  6. The experiments lack statistical significance testing.

  7. The paper should use clearer visualizations to highlight the main experimental findings rather than many tables.

Questions for authors

See Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.