- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how the knowledge density of synthetic data affects LLM knowledge acquisition under a fixed training token budget. The authors define knowledge density to measure data concentration in embedding space and use it to select synthetic training data. Experiments on factual QA benchmarks across different model and data scales show that an appropriate knowledge density can improve knowledge injection.
Critical strengths
The paper addresses an important practical question: how to improve LLM knowledge acquisition under a fixed training token budget. It introduces an explicit measure of semantic concentration and uses it to guide synthetic data selection, providing a concrete way to control data distribution.
Critical weaknesses
-
The main observation that highly concentrated training data leads to worse factual QA performance is reasonably expected. The paper does not sufficiently establish what additional insight or practical value its density measure provides beyond general data diversity and coverage considerations.
-
The training procedure is insufficiently specified. The paper does not clearly state the training objective, the final input–output format, whether knowledge points and logic chains are included in training, or which tokens contribute to the loss. These omissions limit reproducibility and make the knowledge injection results harder to interpret.
Questions for authors
Could the authors provide the prompt templates and target format used for training and evaluation? Are the models explicitly instructed to produce only a short answer, and do the training targets contain any reasoning rationales? In particular, how does the gold-answer loss evaluation account for valid responses that include introductory text or explanations before the answer? This clarification is important for distinguishing factual knowledge acquisition from changes in response-format preferences.