- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GRAFT, which constructs graphs from a model’s internal activations to select data for continual pretraining. The method selects semantic anchors using predecessor-token diversity, represents nodes with their layer-wise MLP up-projection activations, and connects nodes based on cosine similarity. The authors define Branch Fraction (BF) to measure non-isolated node coverage and degree structure, prioritizing documents with low BF scores. The experiments cover three base models, ten knowledge and reasoning tasks, and a mixture of mathematics and STEM corpora. GRAFT achieves high average performance in the reported comparisons. The paper also analyzes graph changes before and after continual pretraining, associating new connections with reasoning improvements and lost connections with forgetting.
Critical strengths
-
The data-selection results show performance gains across different models. In Table 1, GRAFT achieves the highest Macro scores on both SmolLM2-1.7B and Qwen2.5-3B. In Appendix Table 2, its Qwen3-8B Macro score is 82.58%, compared with 81.50% for QuRating and 80.98% for random selection. These comparisons use the same candidate pool and training-token budget. They suggest that the internal activation signal is worth studying and provide the main evidence supporting acceptance.
-
The method offers a data-selection approach distinct from text-quality scores and prediction loss. GRAFT directly uses the target model’s intermediate representations without training an additional scoring model. Layer-wise selection and mixed-domain experiments further examine the conditions under which the signal is useful, extending the contribution beyond a scoring comparison on a single corpus.
Critical weaknesses
-
BF has not been sufficiently validated as an indicator of missing knowledge. Nodes are activations at selected token positions, and edges are defined by activation similarity exceeding a threshold. However, the paper does not sufficiently establish that these connections represent knowledge relations rather than textual repetition, topic consistency, punctuation patterns, or representation-space anisotropy. BF itself depends only on non-isolated node coverage and degree statistics, without using edge directions or path structure. The appendix also acknowledges that it cannot determine whether the entire graph is connected. An increase in BF on the same corpus after training does not rule out general changes in representation geometry. In the mixed-domain experiment, the mathematics proportion also changes from approximately 95% under PPL selection to 55% under GRAFT, so the gains may primarily reflect domain composition. Without controls for these factors, the interpretation of GRAFT as selecting data that fills knowledge gaps remains uncertain.
-
Important implementation and experimental details are missing, limiting reproducibility and confidence in the results. Section 2.3 and Appendix C describe parameter-distribution analyses but do not clearly specify the final anchor-selection settings, similarity thresholds, BF thresholds, voting conditions, or the procedure for selecting documents when eligible data exceed the token budget. Training settings such as learning rate, batch size, and learning-rate schedule are also unspecified, and the main results do not report variation across runs. The efficiency discussion claims that feed-forward computation is performed only at anchor positions, yet deep-layer anchor representations generally depend on computations at other positions in earlier layers. The paper does not explain how this is implemented or clearly define the timing scope. These omissions directly affect assessment of the performance comparisons and efficiency contribution.
Questions for authors
-
What anchor-selection rules, similarity thresholds, BF thresholds, and voting conditions are used for each model in the main experiments? How is the final document set determined, and are any of the reported downstream test tasks used to select parameters?
-
Does BF still predict gains from continual pretraining after controlling for document length, anchor count, domain, and text quality? What evidence distinguishes knowledge familiarity from general activation similarity or textual properties?
-
Does GRAFT retain an advantage when random selection and PPL use the same mathematics/STEM ratio of approximately 55%/45%? This comparison is needed to determine whether the mixed-domain gains come from the BF signal rather than domain composition.
-
How is feed-forward computation restricted to semantic-anchor positions? Does this alter the model’s normal computation at other token positions? Does Figure 4 include the full forward pass, activation extraction, graph construction, and document scoring?
-
Do all methods use identical optimizers, learning rates, sequence lengths, and training schedules? How many runs support the main-table results, and are the gains over strong baselines such as QuRating consistent across runs?
-
Some figures have text and details that are difficult to read, including Figures 1 and 2. Please check all figures throughout the paper and use vector graphics where possible to improve readability.
4 comments
W1b/Q2
- W1b/Q2: Does BF still predict gains from continual pretraining after controlling for document length, anchor count, domain, and text quality? What evidence distinguishes knowledge familiarity from general activation similarity or textual properties?
A: Thanks for your question. GRAFT still predict gainsafter cotrolling for document length, anchor count, domain, and text quality. In experiments, we already use Nemtron-Speciallized-v2.1 data, which is already a high quality data for CPT. Here we give the further analysis on how different methods select the data to train. We analysis it on Qwen2.5-3B based backbone model and formulate it into a. formulation b. document length c. token efficient for domain knowledge.
a. As shown in table, we report the data formulation that labeled by Nemtron-Specialized data. The results suggest GRAFT select the simlar sampling weight campring with the random setting, but others will select more QA formulation. This suggest the data formulation is a minor impact to the final result.
| Method | Fact-Seeking | Multiple-Choice(QA) | Generative | Moral-Scenarios |
|---|---|---|---|---|
| Random | 82.11% | 16.24% | 1.61% | 0.036% |
| DCLM | 88.76% | 10.61% | 0.63% | 0% |
| QuRating | 41.98% | 52.18% | 5.67% | 0.169% |
| PPL | 62.64% | 30.79% | 6.56% | 0.010% |
| Attn-Inf | 43.37% | 50.47% | 6.01% | 0.144% |
| GRAFT | 79.28% | 19.51% | 1.12% | 0.103% |
b. We further report the length of selected documents in the below table. The results show that GRAFT do not have the bias to select data that have more length or short length comparing with others. Meanwhile, the method do not have bias in selecting anchor numbers.
| Metric | Random | DCLM | QuRating | Low PPL | Attn-Inf | GRAFT |
|---|---|---|---|---|---|---|
| Median length(tokens) | 56 | 36 | 86.5 | 102 | 122.5 | 72 |
| Median BF Top30anchor count | 9 | 6 | 17 | 18 | 23 | 12 |
c. We further analysis the domain data distribution and token efficient for each domain. Specifically, we use Qwen3.5-9B to label all of the 10 benchmark with 18 various domain labels, and also for the selected data for each method.
In this way, we got the propotion of domain data selected by various methods and how the selected help increasing accuracy by token efficient \Delta_{acc}/{#token} We report the main reasults are shown in below table.
| Fine-Grained Domain | Random | DCLM | QuRating | Low PPL | Attn-Inf | GRAFT |
|---|---|---|---|---|---|---|
| Law | 3.95%/+12.74 | 2.42%/+12.92 | 3.25%/+18.33 | 2.35%/+20.28 | 2.03%/+25.61 | 4.06%/+13.56 |
| Philosophy,Ethics&Religion | 0.48%/+143.66 | 0.45%/+30.01 | 1.66%/+39.31 | 0.71%/+64.45 | 0.80%/+55.01 | 0.50%/+164.03 |
| Engineering&Manufacturing | 1.29%/+40.38 | 0.81%/+12.91 | 1.50%/+62.81 | 0.60%/+131.51 | 0.63%/+84.93 | 1.61%/+48.50 |
| Chemistry | 1.17%/+36.19 | 1.07%/+1.65 | 3.72%/+24.25 | 3.59%/+13.62 | 3.13%/+5.13 | 0.93%/+80.00 |
| Economics&Finance | 2.53%/+13.47 | 2.88%/+4.53 | 1.62%/+31.33 | 2.83%/+13.11 | 2.12%/+13.42 | 1.98%/+26.42 |
| Business&Management | 0.45%/+53.72 | 0.29%/+35.00 | 0.60%/+76.45 | 0.15%/+205.42 | 0.24%/+144.93 | 0.43%/+99.75 |
| Physics&Astronomy | 2.35%/+18.84 | 1.70%/+6.96 | 5.45%/+14.00 | 4.78%/+12.26 | 4.08%/+6.48 | 2.47%/+29.71 |
| History&Archaeology | 33.79%/+1.19 | 31.26%/−0.20 | 13.08%/+3.99 | 28.59%/+0.96 | 16.59%/+1.95 | 35.34%/+1.50 |
| Medicine&Health | 1.80%/+21.38 | 1.30%/+4.76 | 4.86%/+15.28 | 2.57%/+27.79 | 2.45%/+11.67 | 2.04%/+35.95 |
| Mathematics&Statistics | 8.20%/+1.41 | 7.88%/−0.30 | 10.00%/+4.05 | 17.33%/+0.53 | 13.98%/+0.02 | 7.94%/+7.81 |
| Social&Everyday Situations | 7.69%/+4.94 | 4.18%/+4.94 | 11.17%/+4.20 | 2.26%/+16.24 | 8.07%/+4.90 | 7.47%/+5.91 |
| Psychology | 0.29%/+143.72 | 0.19%/+43.56 | 1.20%/+42.35 | 0.28%/+142.69 | 0.42%/+74.99 | 0.35%/+139.79 |
| Earth&Environment | 2.21%/+28.56 | 4.01%/+3.60 | 2.81%/+30.78 | 3.13%/+24.40 | 2.05%/+21.22 | 2.23%/+39.62 |
| Objects&Space | 9.02%/+4.33 | 14.24%/+1.04 | 3.84%/+12.86 | 6.67%/+6.94 | 6.88%/+5.44 | 6.26%/+8.67 |
| Language&Literature | 14.70%/+3.35 | 12.61%/−0.86 | 19.97%/+3.37 | 10.71%/+3.63 | 24.38%/+1.29 | 16.92%/+3.56 |
| Biology&Ecology | 2.72%/+14.69 | 5.37%/+0.79 | 5.92%/+11.20 | 2.86%/+18.73 | 2.69%/+5.76 | 2.81%/+23.74 |
| Tools&Procedures | 0.61%/+47.47 | 0.12%/+163.34 | 2.66%/+18.40 | 0.50%/+96.28 | 1.73%/+27.88 | 0.76%/+62.03 |
| Computing&Programming | 3.14%/+11.02 | 4.93%/+0.88 | 6.67%/+10.90 | 6.06%/+9.30 | 5.80%/+3.98 | 2.40%/+25.71 |
The results suggest that all of method will chooce various of domains, but GRAFT will choose the most efficient tokens to help the model learn the specific tasks. GRAFT take the first place in token effient in 8 domains and 4 in second. This suggest that GRAFT is able to equally select the most efficient data for as much domains as possible. We will add the analysis details in Appendix F.1.
W1d/Q3/W2a/Q1
- W1d/Q3: Does GRAFT retain an advantage when random selection and PPL use the same mathematics/STEM ratio of approximately 55%/45%?
A: Thank you for your question. We would like to clarify that this ratio is determined by each model’s scores on a mixed candidate pool instead of manually. As noted in line 320, PPL produces an imbalanced score distribution. We additionally report an alternative setting in which we manually control this ratio during continual pretraining on 6B tokens:
| Method | GSM8K | MATH | SVAMP | MAWPS | TheoremQA | GSM-Hard | ASDiv | Macro Acc. |
|---|---|---|---|---|---|---|---|---|
| BF-at-PPL-ratio | 74.48 ± 0.46 | 47.52 ± 0.16 | 81.63 ± 0.15 | 71.73 ± 0.33 | 31.71 ± 0.69 | 37.83 ± 0.72 | 79.73 ± 0.32 | 60.66 ± 0.14 |
| PPL-at-BF-ratio | 71.57 ± 0.73 | 49.61 ± 0.34 | 71.33 ± 0.35 | 70.19 ± 0.38 | 32.92 ± 0.94 | 37.96 ± 0.46 | 83.89 ± 0.07 | 59.64 ± 0.15 |
| method | MMLU | ARC-E | ARC-C | CSQA | QASC | MMLU-Pro | PIQA | OpenBookQA | AGIEval-SAT | HellaSwag | Macro |
|---|---|---|---|---|---|---|---|---|---|---|---|
| BF-at-PPL-ratio | 56.55±0.05 | 89.44±0.11 | 79.78±0.17 | 65.38±0.13 | 66.59±0.16 | 40.01±0.06 | 78.82±0.14 | 72.27±1.15 | 54.32±0.90 | 60.58±0.15 | 66.37±0.02 |
| PPL-at-BF-ratio | 30.17±0.06 | 34.03±0.02 | 33.53±0.23 | 22.25±0.13 | 20.52±0.00 | 42.71±0.07 | 55.97±0.13 | 36.80±1.59 | 36.55±0.47 | 44.17±0.08 | 35.67±0.13 |
The results suggest that BF method still keep the high performance with the PPL manually ratio.
W2a/Q1: parameter setting
For all of experiments, we set (K=30, τ=0.7) for each setting. The training parameters are shown below. We've added more details in Section G.
| Hyperparameter | Qwen2.5-3B/SmolLM2-1.7B | Qwen3-8B-Base(1B-token runs) |
|---|---|---|
| Sequence length | 2,048tokens | 2,048tokens |
| Global batch size | 256sequences | 512sequences |
| Optimizer | AdamW | AdamW |
| AdamW$(\beta_1,\beta_2)$ | (0.9,0.95)\ | |
| Weight decay | ||
| Peak learning rate | ||
| Learning rate schedule | Cosine decay | Cosine decay |
| Warmup ratio | 0.5% | 0.5% |
| Minimum learning rate |
- W2b: variation runs
A: Thanks for the comments. We've updated the random seed (3 runs) in Table 1 and Table 2 to show the variance of each methods.
| Method | MMLU | ARC-E | ARC-C | CSQA | QASC | MMLU-Pro | PIQA | OpenBook | AGIEval | Hella | Macro | Avg.Rank |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Base | 43.93±0.04 | 69.16±0.15 | 57.85±0.09 | 48.46±0.09 | 44.85±0.16 | 32.27±0.09 | 67.30±0.19 | 61.13±1.55 | 57.38±0.24 | 57.66±0.28 | 54.00±0.12 | 5.70 |
| Random | 55.17±0.05 | 80.51±0.15 | 69.48±0.13 | 65.93±0.08 | 60.01±0.41 | 41.58±0.12 | 76.24±0.16 | 71.60±1.00 | 44.88±0.97 | 53.80±0.37 | 61.92±0.07 | 4.30 |
| DCLM | 43.81±0.02 | 64.73±0.11 | 58.87±0.09 | 61.37±0.05 | 53.42±0.06 | 38.84±0.12 | 70.89±0.24 | 59.40±1.74 | 39.24±0.16 | 52.49±0.29 | 54.31±0.15 | 6.50 |
| QuRating | 62.09±0.07 | 92.44±0.09 | 81.80±0.18 | 71.09±0.14 | 71.89±0.12 | 42.75±0.07 | 78.40±0.30 | 81.60±0.00 | 49.58±0.51 | 57.68±0.15 | 68.93±0.05 | 2.00 |
| PPL | 53.61±0.10 | 82.67±0.11 | 74.54±0.18 | 68.50±0.13 | 68.14±0.29 | 42.29±0.07 | 78.80±0.08 | 79.00±0.69 | 43.72±0.80 | 57.48±0.29 | 64.87±0.10 | 3.40 |
| Attn-Inf | 51.57±0.04 | 68.80±0.02 | 64.39±0.26 | 64.84±0.13 | 55.29±0.60 | 43.21±0.02 | 78.78±0.20 | 65.40±1.04 | 41.88±0.97 | 56.38±0.21 | 59.05±0.13 | 4.70 |
| GRAFT | 62.63±0.05 | 91.67±0.04 | 81.11±0.26 | 72.07±0.08 | 70.48±0.06 | 43.60±0.04 | 79.63±0.14 | 81.13±0.83 | 58.18±0.78 | 59.83±0.05 | 70.03±0.06 | 1.40 |
| Method | MMLU | ARC-E | ARC-C | CSQA | QASC | MMLU-Pro | PIQA | OBQA | AGIEval | Hella | Macro | Avg.Rank↓ | Δ Base |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Base | 69.02±0.09 | 92.76±0.08 | 88.65±0.09 | 79.42±0.17 | 79.27±0.39 | 56.13±0.09 | 83.22±0.11 | 87.93±0.50 | 73.52±0.87 | 77.19±0.20 | 78.71±0.10 | 4.50 | — |
| Random | 75.35±0.01 | 96.93±0.04 | 92.58±0.15 | 80.51±0.00 | 81.25±0.16 | 59.95±0.08 | 86.42±0.30 | 89.13±0.31 | 76.74±0.27 | 69.48±0.16 | 80.83±0.00 | 2.90 | +2.12±0.11 |
| DCLM | 72.93±0.07 | 97.14±0.04 | 92.63±0.10 | 78.95±0.00 | 80.31±0.06 | 63.30±0.05 | 88.05±0.08 | 89.00±0.20 | 78.59±0.40 | 68.89±0.16 | 80.98±0.05 | 3.10 | +2.27±0.06 |
| QuRating | 76.16±0.07 | 96.63±0.04 | 91.64±0.09 | 79.69±0.08 | 83.01±0.16 | 64.29±0.07 | 86.76±0.13 | 90.53±0.76 | 76.48±0.80 | 69.13±0.46 | 81.43±0.16 | 2.90 | +2.72±0.07 |
| GRAFT | 76.47±0.09 | 97.28±0.06 | 92.15±0.09 | 80.32±0.13 | 83.12±0.06 | 64.47±0.09 | 86.80±0.13 | 90.13±1.21 | 79.48±0.51 | 74.51±0.16 | 82.47±0.10 | 1.60 | +3.76±0.19 |
W2C/Q4
- W2c/Q4: efficient discussion
A: Thank you for your question. We will provide a more detailed efficiency analysis and clarify that the reported timings cover GPU computation. The top-K token analysis is performed once on the CPU using a sample of 10,000 documents and takes 0.234 seconds. The resulting anchor set is reused across the entire dataset through masking during the forward pass. This one-time preprocessing cost is therefore negligible when amortized over a large dataset. GRAFT benefits from avoiding vocabulary-wide loss computation, particularly when the tokenizer’s vocabulary is large. During the forward pass, the input consists of a token sequence and a corresponding mask identifying the anchor set, while the model retains its standard causal attention mechanism. In our experiments, we use batch sizes of 356, 32, and 12 for documents containing fewer than 500 tokens, 500–2,000 tokens, and more than 2,000 tokens, respectively.
Below, we provide a breakdown of the main computational components involved in calculating BF and PPL scores for each document.
| Document length(tokens) | Full forward pass and activation extraction | Graph construction | BF reduction |
|---|---|---|---|
| <500 | 5.404 | 0.239 | 0.214 |
| 500–2000 | 37.671 | 0.870 | 0.192 |
| >2000 | 131.808 | 2.496 | 0.228 |
| Document length(tokens) | Backbone forward pass | LM head,cross-entropy and loss reduction |
|---|---|---|
| <500 | 5.451 | 0.213 |
| 500–2000 | 36.959 | 2.683 |
| >2000 | 128.846 | 8.386 |
Therefore, graph construction and score aggregation require less computation than cross-entropy loss calculation. Loss-based methods operate over the full tokenizer vocabulary, which can incur substantial computational costs for backbones with large vocabularies.
- Q6 Fix the images:
A: Thanks for your carefully reading. We've updated the iamges.
W1a/W1c
<0.2[0.2,0.4)[0.4,0.6)≥0.6