- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a method for selecting (or filtering) domain-specific data for continued pretraining LLMs. The proposed method, Frequency-Conceptual Difficulty (FCD), uses the Google Books Ngram dataset to quantify lexical rarity (or frequency) and WordNet to estimate word difficulty by measuring the distance between a word and a set of basic words defined at the A1/A2 levels of the CEFR framework. These word-level scores are then aggregated to produce a document-level difficulty score, where higher scores indicate documents of higher lexical difficulty. Based on these scores, the authors propose three data selection strategies: (1) selecting the hardest examples, (2) sampling examples according to their FCD scores, and (3) curriculum learning, where the model is trained gradually from easier to harder examples. Experiments on two domains, medical and financial, show that continued pretraining on FCD-hard data (i.e., documents with higher lexical difficulty) yields the best performance on a range of medical benchmarks. In contrast, continued pretraining on FCD-easy data performs best on financial test data.
Critical strengths
- Strengths
- The proposed method is interesting and novel in that it leverages long-established lexical resources to guide data selection by quantifying both word frequency and conceptual difficulty.
- The method outperforms several baselines based on domain similarity, diversity, classifier-based selection, and language-model perplexity.
Critical weaknesses
- Weaknesses
- Although the proposed method is interesting, I find the current experimental results difficult to interpret and to derive clear takeaways. I believe additional analysis is needed to explain the observed patterns and provide clearer guidance on when different data selection strategies should be used.
- The main results reported in Table 1 seem to be close to random guessing on some medical benchmarks (e.g., MedQA scores of 34-37; even lower in Table 7).
- I would suggest adding a random baseline for reference, as well as reporting statistical significance tests or standard deviations across multiple runs to show whether the performance differences are meaningful.
- The main experiments are based on two small models (Qwen2.5-0.5B and Llama-3.2-1B). As a result, it is unclear how well the findings generalize to larger models (this has been mentioned in the author's Limitation section).
- However, even within these two model scales, the results are inconsistent. On MedQA, FCD-Hard with Llama underperforms both the original model and standard continued pretraining, whereas the largest improvement is observed with the Qwen model.
- On the medical benchmarks, FCD-hard performs best while FCD-easy performs worst. However, on FinEval, the opposite trend is observed: FCD-easy is best while FCD-hard is worst.
- I am not fully convinced by the explanation provided in Lines 423-429. It does not make super sense to me that
since a pretraining set (YiZhao-FinDataSet) has documents with lower FCD scores than other (MMedC), so focusing on those easier subset (FCD-easy) should be a better option. - My understanding is that these results suggest that some tasks may benefit from continued pretraining on more difficult documents, whereas others may benefit from simpler documents.
- Another observation is that the training settings differ across domains. In the financial experiments, the model is trained for only one epoch and FCD-easy performs best, whereas in the medical experiments, training for three epochs on FCD-hard data yields the strongest results.
- It would be useful to investigate whether the observed differences are driven by the original model (e.g., the extent to which it is already familiar with domain-specific terminology), the characteristics of the continued pretraining dataset, the evaluation task, the training duration, or interactions among these factors.
- I am not fully convinced by the explanation provided in Lines 423-429. It does not make super sense to me that
- The main results reported in Table 1 seem to be close to random guessing on some medical benchmarks (e.g., MedQA scores of 34-37; even lower in Table 7).
- Although the proposed method is interesting, I find the current experimental results difficult to interpret and to derive clear takeaways. I believe additional analysis is needed to explain the observed patterns and provide clearer guidance on when different data selection strategies should be used.
Questions for authors
- Questions and Suggestions
- What happens to novel/rare words that cannot be found in either Google Books Ngram or WordNet? How are the frequency score f(w) in Equation 1 and the MSD score in Equation 2 computed for such words?
- I am not sure that
frequency difficulty(Line 152) is the most appropriate term. I find the description in Line 319 clearer and more intuitive 'The comparison tests whether frequency and conceptual distance provide a useful selection signal for domain adaptation.'. - Section 4.1: It is unclear how the figures were generated. For example, on which datasets were the word frequencies and semantic distances measured?
- Section 4.2 (Lines 241-242): This sentence may need to be rewritten. The citation format is incorrect, and it is hard to understand what is being cited.
- Suggest moving Line 381 to a more appropriate location or deleting it.