acceptodds
Under review as a conference paper at ICLR 2027

CALIBUDGET: Calibration-Guided Source Allocation for Fixed-Budget Mixed-Reasoning Adaptation

Abstract

Fixed-budget adaptation from heterogeneous data sources requires deciding not only how much data to use, but how much exposure each source receives. Size-proportional rules can crowd out small sources, whereas difficulty-only rules can chase noisy estimates or allocate residual budget to nearly saturated pools. We introduce CALIBUDGET, a floor-protected, reliability-aware integer allocator that treats source exposure as an explicit adaptation variable. From small train-internal calibration splits, it combines model need, post-floor availability, and bootstrap stability, then produces exact capacity-respecting quotas without changing the model, objective, or total budget. In a controlled setting combining mathematical and commonsense data, CALIBUDGET improves CommonAvg, FragileAvg, and MacroAvg over validation-error-with-floor, the strongest matched comparator, in all three paired LLaMA-2-7B LoRA+ runs. The respective mean gains are 0.56, 0.46, and 0.41 percentage points (pp). Overall increases by 0.18 pp, whereas MathAvg decreases by 0.20 pp, exposing a coverage–retention boundary rather than a uniform gain. CALIBUDGET changes only 1.14–1.42% of the source budget but improves performance in 15 of 24 comparisons across commonsense tasks and seeds. These results suggest that small changes in source quotas can matter; example-level selection can then determine which examples fill each quota. The code is available at https://anonymous.4open.science/r/calibudget-anonymous-code-7470/

Primary area:
foundation or frontier models, including LLMs
Keywords:
source allocation, data mixture, instruction tuning, large language models, fixed-budget adaptation, data selection, multi-source learning, parameter-efficient fine-tuning, LoRA, calibration-guided allocation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_aQhrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces CALIBUDGET, a method that chooses how many training examples to take from each source for a fixed budget. Each source gets a minimum share, and the rest depends on model errors, how stable those estimates are, and how much data is available. The experiments keep the math training data fixed and vary the mix of eight commonsense datasets. Across three LLaMA-2 runs, the method gives a small average improvement on commonsense tasks over the strongest tested baseline, with a small average drop on math tasks. Results on other models are mixed.

Critical strengths
  1. The method improves average commonsense accuracy over the strongest tested baseline consistently in all three main runs.

  2. The proposed stability term seems interesting, as removing this term lowers performance.

Critical weaknesses
  1. Choosing how much data to take from each source is an established problem. The main addition is a particular scoring rule with minimum counts and rounding, and the paper does not clearly show that this is a substantial advance.

  2. The presented results provide limited evidence of a useful improvement. Against the strongest tested baseline, commonsense accuracy improves by 0.56 pp, and the Overall score improves by 0.18 points, while math accuracy drops.

  3. The writing is difficult to follow, unclear, and makes design choices hard to assess. The paper repeats many points but leaves some important choices unclear. For example, it does not state the exact rule used to stop the experiment with a lower minimum number of examples per source. This makes the results harder to assess and reproduce.

  4. At least two references have incorrect author lists: "PIQA", and "Not All Tokens Are What You Need for Pretraining" omit authors from the author lists. These errors suggest that the references need more careful checking, and that the manuscript was not prepared to professional academic standards.

Questions for authors
  1. How were the minimum counts and scoring settings chosen? What exact rule was used to stop the experiment with lower minimum counts?

  2. What does CALIBUDGET add beyond existing methods such as for example SMART (Renduchintala et al., 2024)? Do you have results comparing them under the same data budget and training setup?

SMART: Submodular Data Mixture Strategy for Instruction Tuning (Renduchintala et al., Findings 2024)

Reviewer_1nHxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper tackles the problem of fixed-budget allocation during adaptation in the presence of multiple sources. Given different signals from different sources in terms of need, availability, and reliability, the paper allocates budget for each source as an integer value. Leveraging the exact integer budget from individual sources, the adaptation is performed.

Critical strengths

The problem of having to allocate fine-grained budget while performing adaptation is important, in the presence of multiple sources.

Critical weaknesses
  • What’s the rationale behind using a fixed set as a mathematical training subset rather than a generic one? Also, is the method generalizable if we choose a different set of datasets (both fixed and source-specific)?
  • In lines 142- 143, what does the paper mean by teacher? It is suggested to include a clear description of what the architecture is before explaining the method (example: teacher, task-specific parser).
  • What is the intuition behind measuring aka_k, and how does it represent a source’s residual-availability signal? The design motivation is not clear.
  • There is no description of other signals (calibration need and reliability).
Questions for authors

Please refer to weakness section

Reviewer_JPUkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to split a fixed data budget across several training sources. In the experiments, LLaMA-2-7B is fine-tuned with LoRA+ on 100k MetaMath examples plus 10k commonsense examples drawn from the training sets of eight commonsense benchmarks. CALIBUDGET first gives each source at least 1,150 examples (9,069 in total, since one source is smaller), then splits the remaining 931 examples based on how poorly the model does on a small held-out set from each source, how much data is left, and how reliable that estimate is. Over three seeds, compared with the strongest baseline (which splits the same 931 examples by error alone), it raises the commonsense average by 0.56 pp and lowers the math average by 0.20 pp. On Qwen2.5-7B and LLaMA-3.1-8B the differences are not significant.

Critical strengths
  1. The question is practical. When the training budget is fixed, deciding how many examples to take from each source is a real decision.

  2. The paper is open about its limits. Table 6 lists what the evidence does and does not support.

Critical weaknesses

W1. The gains are small and may be noise, and the paper's own data suggests they do not come from the allocation. The gains over the strongest baseline are 0.18-0.56 pp over three seeds, with no standard deviations in Table 1 and no significance test. The two methods differ by only 114-142 examples (p.7), about 0.1% of the 110k training examples. The math average drops by 0.20 pp, and for seed 42 the overall score is lower than the baseline (Table 2). The paper itself finds that whether a task improves has nothing to do with how many more examples its source received (Pearson r = -0.005; p.7). For example, ARC-Challenge has only 1,019 training examples available (supplement code), fewer than the 1,150 minimum, so both methods use all of them. Its training data is exactly the same under both methods, yet its accuracy still changes by +0.60, +0.94 and +0.34 pp (Figure 2c). To know whether a 0.5 pp gain means anything, the paper needs to show how much the results change when the quotas stay the same and only a different random set of examples is drawn.

W2. The effect does not carry over to newer models. The main result is on LLaMA-2-7B, a 2023 model. On Qwen2.5-7B none of the differences is significant (p.8). On LLaMA-3.1-8B the two methods score about the same (Overall 71.84 vs. 71.88), and the baseline is switched from the strongest one (Val-error+floor) to a weaker one (Floor-Sqrt) without explanation (Table 4). On a set of newer tasks (BBH, HumanEval, IFEval and others), none of the differences is significant either, and there is no difference at all on the coding tasks (p.8). So the gain shows up only on the oldest model, and it is unclear whether it helps current models.

W3. No comparison with existing data-mixture methods. The paper does not compare with DoReMi, RegMix or Data Mixing Laws, saying they output mixing ratios rather than exact example counts (p.1, p.6). But CALIBUDGET itself first computes ratios and then rounds them to whole numbers (p.4), so the ratios from these methods could be rounded the same way and compared directly. The supplement also includes a LESS-style allocation script (build_less_style_proxy_allocation.py), but no LESS results are reported. All baselines are simple rules (size-based, equal, square-root, error-based), so it is hard to tell whether the method improves on existing work.

Questions for authors

Q1. Please see W1-W3. In particular: how much do the results change if the quotas are kept fixed and only the sampled examples or the training seed change? And why does ARC-Challenge improve in every seed when its training data is the same under both methods?

Q2. The paper says test results are not used to choose quotas or hyperparameters (p.9). But the run with a lower minimum (950 instead of 1,150) was stopped because "absolute performance falls below the prespecified continuation criterion" (p.8), and this criterion is never defined. Also, the v2 allocation script in the supplement directly names the four test tasks used in FragileAvg (FRAGILE_LABELS), and gives an extra boost to three sources picked "from prior allocation contrast", that is, from earlier results. The paper only says these variants "alter source normalization" (p.7). How were the exponents (1, 0.5, 1) and the four FragileAvg tasks chosen?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.