acceptodds
Withdrawn from ICLR 2027

WordNet-Guided Domain Data Selection for LLM Continual Pre-training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Continual pre-training (CPT) adapts language models to specialized domains, but selecting useful training data from large domain corpora remains difficult. We introduce Frequency-Conceptual Difficulty (FCD), a model-free metric that combines lexical frequency with semantic distance in WordNet to score document difficulty. In the medical domain, FCD-Hard selects a fixed one-third subset and trains on it for three epochs. Under the same total token exposure, this strategy improves domain-average accuracy over full-corpus CPT by 1.36 percentage points and outperforms the compared data-selection baselines. We also examine whether the preferred FCD range is domain dependent. Medical performance is highest when training on documents with rare, WordNet-distant terminology, whereas FCD-Easy performs best in our experiment on an English translation of FinEval. Qualitative examples suggest that some high-FCD documents in the financial corpus contain technical vocabulary from other domains. These results support using FCD as a lexical criterion for domain-data analysis while showing that the sampling direction must be selected for the target corpus.

Primary area:
foundation or frontier models, including LLMs
Keywords:
data selection, continual pre-training, large language models, domain adaptation, WordNet, data-centric AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uxEdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method for selecting (or filtering) domain-specific data for continued pretraining LLMs. The proposed method, Frequency-Conceptual Difficulty (FCD), uses the Google Books Ngram dataset to quantify lexical rarity (or frequency) and WordNet to estimate word difficulty by measuring the distance between a word and a set of basic words defined at the A1/A2 levels of the CEFR framework. These word-level scores are then aggregated to produce a document-level difficulty score, where higher scores indicate documents of higher lexical difficulty. Based on these scores, the authors propose three data selection strategies: (1) selecting the hardest examples, (2) sampling examples according to their FCD scores, and (3) curriculum learning, where the model is trained gradually from easier to harder examples. Experiments on two domains, medical and financial, show that continued pretraining on FCD-hard data (i.e., documents with higher lexical difficulty) yields the best performance on a range of medical benchmarks. In contrast, continued pretraining on FCD-easy data performs best on financial test data.

Critical strengths
  • Strengths
    • The proposed method is interesting and novel in that it leverages long-established lexical resources to guide data selection by quantifying both word frequency and conceptual difficulty.
    • The method outperforms several baselines based on domain similarity, diversity, classifier-based selection, and language-model perplexity.
Critical weaknesses
  • Weaknesses
    • Although the proposed method is interesting, I find the current experimental results difficult to interpret and to derive clear takeaways. I believe additional analysis is needed to explain the observed patterns and provide clearer guidance on when different data selection strategies should be used.
      • The main results reported in Table 1 seem to be close to random guessing on some medical benchmarks (e.g., MedQA scores of 34-37; even lower in Table 7).
        • I would suggest adding a random baseline for reference, as well as reporting statistical significance tests or standard deviations across multiple runs to show whether the performance differences are meaningful.
      • The main experiments are based on two small models (Qwen2.5-0.5B and Llama-3.2-1B). As a result, it is unclear how well the findings generalize to larger models (this has been mentioned in the author's Limitation section).
        • However, even within these two model scales, the results are inconsistent. On MedQA, FCD-Hard with Llama underperforms both the original model and standard continued pretraining, whereas the largest improvement is observed with the Qwen model.
      • On the medical benchmarks, FCD-hard performs best while FCD-easy performs worst. However, on FinEval, the opposite trend is observed: FCD-easy is best while FCD-hard is worst.
        • I am not fully convinced by the explanation provided in Lines 423-429. It does not make super sense to me that since a pretraining set (YiZhao-FinDataSet) has documents with lower FCD scores than other (MMedC), so focusing on those easier subset (FCD-easy) should be a better option.
        • My understanding is that these results suggest that some tasks may benefit from continued pretraining on more difficult documents, whereas others may benefit from simpler documents.
        • Another observation is that the training settings differ across domains. In the financial experiments, the model is trained for only one epoch and FCD-easy performs best, whereas in the medical experiments, training for three epochs on FCD-hard data yields the strongest results.
        • It would be useful to investigate whether the observed differences are driven by the original model (e.g., the extent to which it is already familiar with domain-specific terminology), the characteristics of the continued pretraining dataset, the evaluation task, the training duration, or interactions among these factors.
Questions for authors
  • Questions and Suggestions
    • What happens to novel/rare words that cannot be found in either Google Books Ngram or WordNet? How are the frequency score f(w) in Equation 1 and the MSD score in Equation 2 computed for such words?
    • I am not sure that frequency difficulty (Line 152) is the most appropriate term. I find the description in Line 319 clearer and more intuitive 'The comparison tests whether frequency and conceptual distance provide a useful selection signal for domain adaptation.'.
    • Section 4.1: It is unclear how the figures were generated. For example, on which datasets were the word frequencies and semantic distances measured?
    • Section 4.2 (Lines 241-242): This sentence may need to be rewritten. The citation format is incorrect, and it is hard to understand what is being cited.
    • Suggest moving Line 381 to a more appropriate location or deleting it.
Reviewer_WetsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies data selection for LLM continual pre-training and proposes Frequency-Conceptual Difficulty (FCD), a model-free lexical score. FCD combines word frequency with distance from basic concepts in WordNet to compute document scores. Based on this metric, the authors develop three strategies: FCD-Hard, FCD-Soft, and FCD-CL. Experiments across several benchmarks support the usefulness of FCD for domain adaptation. However, the medical and financial experiments favor different sampling directions, and it remains unclear which domains each strategy suits and what rules should guide the choice.

Critical strengths
  1. FCD provides a practical approach to scoring and selecting large corpora without running an auxiliary language model. The authors report that scoring a 6B-token corpus requires only 41.6 CPU-hours, suggesting that the scoring procedure is computationally efficient.

  2. The medical results are encouraging. FCD-Hard outperforms the four selection baselines under the same sampled-CPT protocol, and the improvement in domain-average accuracy also appears on Llama-3.2-1B. The paper provides substantial hyperparameter sensitivity analysis to examine how parameter choices affect data selection.

  3. Combining word frequency with WordNet distance gives the selection score a clear and interpretable basis.

Critical weaknesses
  1. The CPT experiments on the English MMedC corpus lack a matched random-subset baseline. FCD-Hard trains on a one-third subset for three epochs, while Base CPT trains on the full corpus for one epoch. Although the total token budget is matched, this comparison does not separate the benefit of FCD selection from that of repeatedly training on a smaller corpus. A random one-third subset trained for three epochs would be an important control for isolating the effect of the selection criterion.

  2. The criterion for choosing the sampling direction remains unclear. FCD-Hard performs best in the medical experiments, whereas FCD-Easy performs best in the financial experiment. Although the paper acknowledges that the appropriate strategy depends on the target domain, it does not establish a concrete selection rule supported by theoretical reasoning and experimental validation.

  3. The efficiency advantage over other selection methods is not directly evaluated. The paper reports the CPU-hours required by FCD, but does not compare scoring time or resource usage with the selection baselines under a common hardware setup. Since low computational cost is a main motivation for the method, this comparison would help assess the trade-off between selection cost and downstream performance.

  4. The fonts in Tables 2 and 4 are too small for comfortable reading. The table and figure arrangements also leave substantial unused space on these pages. Larger fonts and an improved layout would make the results easier to read.

  5. The method is under-determined by theory, and the technical contribution is essentially a set of empirical heuristics. What are the relationship among FCD-CL, FCD-soft and FCD-hard. Why does the curriculum learning perform worse than hard sampling when combined with FCD?

Questions for authors
  1. The paper suggests using different sampling strategies for different target domains. What are the specific selection criteria?

  2. How does FCD compare with the other selection methods in scoring time and computational resource requirements under the same hardware conditions?

Reviewer_TM5XOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Frequency-Conceptual Difficulty (FCD), a model-free method for selecting continual pre-training data using lexical rarity and WordNet-based semantic distance. Experiments on medical and finance tasks show that difficulty-based selection can improve downstream performance, though results vary by domain and the evaluation lacks sufficient statistical validation.

Critical strengths

A lightweight, model-free data-selection criterion. FCD combines corpus frequency with WordNet-based semantic distance and can score large corpora without running an auxiliary language model. This offers a practical alternative to model-based scoring when computational cost is a concern.

Promising results with component-level analysis. FCD-Hard improves the medical benchmark average from 36.87% to 38.23% under the reported equal-token-budget comparison. The ablations also show that removing either frequency or conceptual distance reduces the average accuracy. These results provide initial evidence that the proposed signal is useful for medical CPT.

Critical weaknesses

Insufficient controls for the main performance claim. The medical experiments compare FCD-Hard, which trains for three epochs on one-third of the corpus, against full-corpus CPT trained for one epoch. Although total token exposure is matched, the amount of unique data and number of epochs differ. The medical comparison lacks a random one-third subset trained for three epochs, so the benefit attributable specifically to FCD-based selection is not isolated. This is central to the paper's main claim.

Limited evidence for robustness and cross-domain generality. Most configurations are run once, and the reported gains lack uncertainty estimates. The financial experiment also uses an automatically translated version of FinEval and finds that random sampling already performs strongly (41.84%), while FCD-Easy reaches 43.26%. This result is suggestive, but without repeated runs or stronger controls, it is difficult to establish that the difference is reliable or that FCD generalizes across domains. The small model sizes and limited domain coverage further constrain the conclusions.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.