- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces CALIBUDGET, a method that chooses how many training examples to take from each source for a fixed budget. Each source gets a minimum share, and the rest depends on model errors, how stable those estimates are, and how much data is available. The experiments keep the math training data fixed and vary the mix of eight commonsense datasets. Across three LLaMA-2 runs, the method gives a small average improvement on commonsense tasks over the strongest tested baseline, with a small average drop on math tasks. Results on other models are mixed.
Critical strengths
-
The method improves average commonsense accuracy over the strongest tested baseline consistently in all three main runs.
-
The proposed stability term seems interesting, as removing this term lowers performance.
Critical weaknesses
-
Choosing how much data to take from each source is an established problem. The main addition is a particular scoring rule with minimum counts and rounding, and the paper does not clearly show that this is a substantial advance.
-
The presented results provide limited evidence of a useful improvement. Against the strongest tested baseline, commonsense accuracy improves by 0.56 pp, and the Overall score improves by 0.18 points, while math accuracy drops.
-
The writing is difficult to follow, unclear, and makes design choices hard to assess. The paper repeats many points but leaves some important choices unclear. For example, it does not state the exact rule used to stop the experiment with a lower minimum number of examples per source. This makes the results harder to assess and reproduce.
-
At least two references have incorrect author lists: "PIQA", and "Not All Tokens Are What You Need for Pretraining" omit authors from the author lists. These errors suggest that the references need more careful checking, and that the manuscript was not prepared to professional academic standards.
Questions for authors
-
How were the minimum counts and scoring settings chosen? What exact rule was used to stop the experiment with lower minimum counts?
-
What does CALIBUDGET add beyond existing methods such as for example SMART (Renduchintala et al., 2024)? Do you have results comparing them under the same data budget and training setup?
SMART: Submodular Data Mixture Strategy for Instruction Tuning (Renduchintala et al., Findings 2024)