acceptodds
Under review as a conference paper at ICLR 2027

GRAFT: Growing Missing Knowledge through Topology-Guided Data Selection

Abstract

Continual pretraining requires selecting data that expands a language model's knowledge without eroding what it has already learned. We introduce a method for constructing latent knowledge as layer-wise directed graphs of concept connections. Building on this construction, we develop GRAFT, a data-selection framework that prioritizes documents with weak concept connectivity in middle layers. Training on the selected data strengthens links among previously isolated concepts and improves downstream performance. Across ten knowledge and reasoning benchmarks and three base models, GRAFT achieves the best overall average rank among the evaluated data-selection methods. In our experiments, GRAFT also achieves higher scoring throughput on long documents than the evaluated model-adaptive baselines. When selecting from a mixture of mathematics and STEM-specialized data, GRAFT improves both general knowledge and mathematical reasoning. Finally, we examine how latent knowledge structures in the middle layers of LLMs relate to reasoning and forgetting: improved reasoning coincides with new connected concepts, whereas forgetting is associated with the loss of established ones.

Primary area:
foundation or frontier models, including LLMs
Keywords:
continual pretraining, data selection, language models, internal representations

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wg3jOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GRAFT, which constructs graphs from a model’s internal activations to select data for continual pretraining. The method selects semantic anchors using predecessor-token diversity, represents nodes with their layer-wise MLP up-projection activations, and connects nodes based on cosine similarity. The authors define Branch Fraction (BF) to measure non-isolated node coverage and degree structure, prioritizing documents with low BF scores. The experiments cover three base models, ten knowledge and reasoning tasks, and a mixture of mathematics and STEM corpora. GRAFT achieves high average performance in the reported comparisons. The paper also analyzes graph changes before and after continual pretraining, associating new connections with reasoning improvements and lost connections with forgetting.

Critical strengths
  1. The data-selection results show performance gains across different models. In Table 1, GRAFT achieves the highest Macro scores on both SmolLM2-1.7B and Qwen2.5-3B. In Appendix Table 2, its Qwen3-8B Macro score is 82.58%, compared with 81.50% for QuRating and 80.98% for random selection. These comparisons use the same candidate pool and training-token budget. They suggest that the internal activation signal is worth studying and provide the main evidence supporting acceptance.

  2. The method offers a data-selection approach distinct from text-quality scores and prediction loss. GRAFT directly uses the target model’s intermediate representations without training an additional scoring model. Layer-wise selection and mixed-domain experiments further examine the conditions under which the signal is useful, extending the contribution beyond a scoring comparison on a single corpus.

Critical weaknesses
  1. BF has not been sufficiently validated as an indicator of missing knowledge. Nodes are activations at selected token positions, and edges are defined by activation similarity exceeding a threshold. However, the paper does not sufficiently establish that these connections represent knowledge relations rather than textual repetition, topic consistency, punctuation patterns, or representation-space anisotropy. BF itself depends only on non-isolated node coverage and degree statistics, without using edge directions or path structure. The appendix also acknowledges that it cannot determine whether the entire graph is connected. An increase in BF on the same corpus after training does not rule out general changes in representation geometry. In the mixed-domain experiment, the mathematics proportion also changes from approximately 95% under PPL selection to 55% under GRAFT, so the gains may primarily reflect domain composition. Without controls for these factors, the interpretation of GRAFT as selecting data that fills knowledge gaps remains uncertain.

  2. Important implementation and experimental details are missing, limiting reproducibility and confidence in the results. Section 2.3 and Appendix C describe parameter-distribution analyses but do not clearly specify the final anchor-selection settings, similarity thresholds, BF thresholds, voting conditions, or the procedure for selecting documents when eligible data exceed the token budget. Training settings such as learning rate, batch size, and learning-rate schedule are also unspecified, and the main results do not report variation across runs. The efficiency discussion claims that feed-forward computation is performed only at anchor positions, yet deep-layer anchor representations generally depend on computations at other positions in earlier layers. The paper does not explain how this is implemented or clearly define the timing scope. These omissions directly affect assessment of the performance comparisons and efficiency contribution.

Questions for authors
  1. What anchor-selection rules, similarity thresholds, BF thresholds, and voting conditions are used for each model in the main experiments? How is the final document set determined, and are any of the reported downstream test tasks used to select parameters?

  2. Does BF still predict gains from continual pretraining after controlling for document length, anchor count, domain, and text quality? What evidence distinguishes knowledge familiarity from general activation similarity or textual properties?

  3. Does GRAFT retain an advantage when random selection and PPL use the same mathematics/STEM ratio of approximately 55%/45%? This comparison is needed to determine whether the mixed-domain gains come from the BF signal rather than domain composition.

  4. How is feed-forward computation restricted to semantic-anchor positions? Does this alter the model’s normal computation at other token positions? Does Figure 4 include the full forward pass, activation extraction, graph construction, and document scoring?

  5. Do all methods use identical optimizers, learning rates, sequence lengths, and training schedules? How many runs support the main-table results, and are the gains over strong baselines such as QuRating consistent across runs?

  6. Some figures have text and details that are difficult to read, including Figures 1 and 2. Please check all figures throughout the paper and use vector graphics where possible to improve readability.

4 comments
  • W1a/W1c: textual repetition, topic consistency, punctuation patterns, or representation-space anisotropy(inner representation geometry).

A: Thanks for your questions. Here we show the relations with the several terms: a. repeatition and punctuation patterns. We apply document quality controls before constructing the candidate pool. At this stage, rule-based filters can screen for obvious text defects. In experiment, we already used the filtered data, where the candidate pool is high quality Nemtron-Specialized-v2.1. Even so, we would like to show BF have the basic ability in distinguishing the repeat token span. We randomly sampled 10,000 documents for which 1–3 layers satisfied the BF<0.1\mathrm{BF}<0.1 criterion. For each document, we constructed two variants of equal token length. In the repetition condition, we repeated a 5–10-token span from the original document. In the control condition, we added the same number of tokens while preserving their original order. For each condition, we averaged BF across the 10,000 documents at each of the 36 layers and counted how many layer means fell into each interval.

Mean BF per layeroriginal BF scoreRepeat a5–10token span
<0.250
[0.2,0.4)1511
[0.4,0.6)516
≥0.6119

Under the tested perturbation, all five layers with a mean BF below 0.2 in the control reached at least 0.2 after repetition. The number of layers with mean BF in [0.4,0.6)[0.4,0.6) also increased from 5 to 16. This shows that BF is able to distinguish the repeated spans. b. topic consistency: Topic relatedness naturally contributes to contextual concept representations, and knowledge integration can involve connections both within and across topics. In particular, acquiring new knowledge may establish a previously missing connection between concepts from different domains. GRAFT aims to identify such gaps relative to the model’s existing representations. c. representation-space anisotropy: We conducted an additional analysis on held-out data to examine domain effects. We grouped the subdomains into two broad categories: STEM and humanities (history, philosophy, psychology, economics, business, law, and language). For each category, we selected 0.3B tokens for continual pretraining (CPT). We then measured layer-wise changes in BF scores relative to the base model on held-out documents from both categories. The table below reports the top five layers with the largest changes. Here, “A CPT × B” denotes continual pretraining on domain A and evaluation on domain B.

LayerHumanities CPT×HumanitiesHumanities CPT×STEMSTEM CPT×HumanitiesSTEM CPT×STEM
L20+90.44%+81.60%+75.50%+84.21%
L21+77.63%+70.20%+67.38%+68.10%
L22+55.52%+51.69%+46.46%+48.49%
L24+50.98%+43.69%+46.05%+43.95%
L23+46.10%+41.47%+41.13%+42.00%
Top-5mean·All documents with valid scores+64.13%+57.73%+55.31%+57.35%

Across these top five layers, the average increase in BF scores is larger when the evaluation domain matches the CPT domain: 64.13% versus 57.73% for humanities CPT, and 57.35% versus 55.31% for STEM CPT. These results suggest that continual pretraining has a stronger effect on BF scores for documents from the same domain.

  • W1b. The ablation of toplogical structures.

A: Thank you for raising this distinction. BF’s many-to-one mapping is a deliberate design choice. By combining non-isolated node coverage and degree structure, BF places diverse sparse connectivity patterns on a common scale for data selection. Appendix A.2 and Figure 15 illustrate this flexibility with structurally distinct graphs that receive the same BF score. Further, We evaluated the BF components and anchor selection on the same 10,000 documents using Qwen2.5-3B Base and using threshold of τ=0.70\tau=0.70. We computed both metrics separately at each layer and averaged them equally across all 36 layers. H50=(1−∑bpb2)/(1−1/50)H_{50}=(1-\sum_b p_b^2)/(1-1/50) measures normalized score diversity, where pbp_b is the fraction of documents in bin bb and bin boundaries are defined by pooled score quantiles. The exact tie rate is the fraction of distinct document pairs with identical scores, including pairs where both scores are zero.

ScoreMean H50↑Exact tie rate↓
Non-isolated0.803919.12%
Degree0.807318.16%
BF0.815016.80%

Under the same graph construction, combining the two terms yields the highest score diversity and reduces exact ties by 1.36 percentage points relative to the degree term, the stronger of the two individual components. Since our backbone LLMs use causal attention, graph edges consistently point from left to right in token order. This fixed ordering simplifies BF computation, eliminating the need to model edge direction separately. We also formulate BF scoring using GPU matrix operations, avoiding explicit computation over complex path structures that would introduce additional overhead.

  • W1b/Q2: Does BF still predict gains from continual pretraining after controlling for document length, anchor count, domain, and text quality? What evidence distinguishes knowledge familiarity from general activation similarity or textual properties?

A: Thanks for your question. GRAFT still predict gainsafter cotrolling for document length, anchor count, domain, and text quality. In experiments, we already use Nemtron-Speciallized-v2.1 data, which is already a high quality data for CPT. Here we give the further analysis on how different methods select the data to train. We analysis it on Qwen2.5-3B based backbone model and formulate it into a. formulation b. document length c. token efficient for domain knowledge.

a. As shown in table, we report the data formulation that labeled by Nemtron-Specialized data. The results suggest GRAFT select the simlar sampling weight campring with the random setting, but others will select more QA formulation. This suggest the data formulation is a minor impact to the final result.

MethodFact-SeekingMultiple-Choice(QA)GenerativeMoral-Scenarios
Random82.11%16.24%1.61%0.036%
DCLM88.76%10.61%0.63%0%
QuRating41.98%52.18%5.67%0.169%
PPL62.64%30.79%6.56%0.010%
Attn-Inf43.37%50.47%6.01%0.144%
GRAFT79.28%19.51%1.12%0.103%

b. We further report the length of selected documents in the below table. The results show that GRAFT do not have the bias to select data that have more length or short length comparing with others. Meanwhile, the method do not have bias in selecting anchor numbers.

MetricRandomDCLMQuRatingLow PPLAttn-InfGRAFT
Median length(tokens)563686.5102122.572
Median BF Top30anchor count9617182312

c. We further analysis the domain data distribution and token efficient for each domain. Specifically, we use Qwen3.5-9B to label all of the 10 benchmark with 18 various domain labels, and also for the selected data for each method.

In this way, we got the propotion of domain data selected by various methods and how the selected help increasing accuracy by token efficient \Delta_{acc}/{#token} We report the main reasults are shown in below table.

Fine-Grained DomainRandomDCLMQuRatingLow PPLAttn-InfGRAFT
Law3.95%/+12.742.42%/+12.923.25%/+18.332.35%/+20.282.03%/+25.614.06%/+13.56
Philosophy,Ethics&Religion0.48%/+143.660.45%/+30.011.66%/+39.310.71%/+64.450.80%/+55.010.50%/+164.03
Engineering&Manufacturing1.29%/+40.380.81%/+12.911.50%/+62.810.60%/+131.510.63%/+84.931.61%/+48.50
Chemistry1.17%/+36.191.07%/+1.653.72%/+24.253.59%/+13.623.13%/+5.130.93%/+80.00
Economics&Finance2.53%/+13.472.88%/+4.531.62%/+31.332.83%/+13.112.12%/+13.421.98%/+26.42
Business&Management0.45%/+53.720.29%/+35.000.60%/+76.450.15%/+205.420.24%/+144.930.43%/+99.75
Physics&Astronomy2.35%/+18.841.70%/+6.965.45%/+14.004.78%/+12.264.08%/+6.482.47%/+29.71
History&Archaeology33.79%/+1.1931.26%/−0.2013.08%/+3.9928.59%/+0.9616.59%/+1.9535.34%/+1.50
Medicine&Health1.80%/+21.381.30%/+4.764.86%/+15.282.57%/+27.792.45%/+11.672.04%/+35.95
Mathematics&Statistics8.20%/+1.417.88%/−0.3010.00%/+4.0517.33%/+0.5313.98%/+0.027.94%/+7.81
Social&Everyday Situations7.69%/+4.944.18%/+4.9411.17%/+4.202.26%/+16.248.07%/+4.907.47%/+5.91
Psychology0.29%/+143.720.19%/+43.561.20%/+42.350.28%/+142.690.42%/+74.990.35%/+139.79
Earth&Environment2.21%/+28.564.01%/+3.602.81%/+30.783.13%/+24.402.05%/+21.222.23%/+39.62
Objects&Space9.02%/+4.3314.24%/+1.043.84%/+12.866.67%/+6.946.88%/+5.446.26%/+8.67
Language&Literature14.70%/+3.3512.61%/−0.8619.97%/+3.3710.71%/+3.6324.38%/+1.2916.92%/+3.56
Biology&Ecology2.72%/+14.695.37%/+0.795.92%/+11.202.86%/+18.732.69%/+5.762.81%/+23.74
Tools&Procedures0.61%/+47.470.12%/+163.342.66%/+18.400.50%/+96.281.73%/+27.880.76%/+62.03
Computing&Programming3.14%/+11.024.93%/+0.886.67%/+10.906.06%/+9.305.80%/+3.982.40%/+25.71

The results suggest that all of method will chooce various of domains, but GRAFT will choose the most efficient tokens to help the model learn the specific tasks. GRAFT take the first place in token effient in 8 domains and 4 in second. This suggest that GRAFT is able to equally select the most efficient data for as much domains as possible. We will add the analysis details in Appendix F.1.

AuthorsOpenReview ↗

W1d/Q3/W2a/Q1

  • W1d/Q3: Does GRAFT retain an advantage when random selection and PPL use the same mathematics/STEM ratio of approximately 55%/45%?

A: Thank you for your question. We would like to clarify that this ratio is determined by each model’s scores on a mixed candidate pool instead of manually. As noted in line 320, PPL produces an imbalanced score distribution. We additionally report an alternative setting in which we manually control this ratio during continual pretraining on 6B tokens:

MethodGSM8KMATHSVAMPMAWPSTheoremQAGSM-HardASDivMacro Acc.
BF-at-PPL-ratio74.48 ± 0.4647.52 ± 0.1681.63 ± 0.1571.73 ± 0.3331.71 ± 0.6937.83 ± 0.7279.73 ± 0.3260.66 ± 0.14
PPL-at-BF-ratio71.57 ± 0.7349.61 ± 0.3471.33 ± 0.3570.19 ± 0.3832.92 ± 0.9437.96 ± 0.4683.89 ± 0.0759.64 ± 0.15
methodMMLUARC-EARC-CCSQAQASCMMLU-ProPIQAOpenBookQAAGIEval-SATHellaSwagMacro
BF-at-PPL-ratio56.55±0.0589.44±0.1179.78±0.1765.38±0.1366.59±0.1640.01±0.0678.82±0.1472.27±1.1554.32±0.9060.58±0.1566.37±0.02
PPL-at-BF-ratio30.17±0.0634.03±0.0233.53±0.2322.25±0.1320.52±0.0042.71±0.0755.97±0.1336.80±1.5936.55±0.4744.17±0.0835.67±0.13

The results suggest that BF method still keep the high performance with the PPL manually ratio.

W2a/Q1: parameter setting

For all of experiments, we set (K=30, τ=0.7) for each setting. The training parameters are shown below. We've added more details in Section G.

HyperparameterQwen2.5-3B/SmolLM2-1.7BQwen3-8B-Base(1B-token runs)
Sequence length2,048tokens2,048tokens
Global batch size256sequences512sequences
OptimizerAdamWAdamW
AdamW$(\beta_1,\beta_2)$(0.9,0.95)(0.9,0.95)(0.9,0.95)\
Weight decay0.10.10.10.1
Peak learning rate1×10−51\times10^{-5}5×10−65\times10^{-6}
Learning rate scheduleCosine decayCosine decay
Warmup ratio0.5%0.5%
Minimum learning rate1×10−61\times10^{-6}5×10−75\times10^{-7}
  • W2b: variation runs

A: Thanks for the comments. We've updated the random seed (3 runs) in Table 1 and Table 2 to show the variance of each methods.

MethodMMLUARC-EARC-CCSQAQASCMMLU-ProPIQAOpenBookAGIEvalHellaMacroAvg.Rank
Base43.93±0.0469.16±0.1557.85±0.0948.46±0.0944.85±0.1632.27±0.0967.30±0.1961.13±1.5557.38±0.2457.66±0.2854.00±0.125.70
Random55.17±0.0580.51±0.1569.48±0.1365.93±0.0860.01±0.4141.58±0.1276.24±0.1671.60±1.0044.88±0.9753.80±0.3761.92±0.074.30
DCLM43.81±0.0264.73±0.1158.87±0.0961.37±0.0553.42±0.0638.84±0.1270.89±0.2459.40±1.7439.24±0.1652.49±0.2954.31±0.156.50
QuRating62.09±0.0792.44±0.0981.80±0.1871.09±0.1471.89±0.1242.75±0.0778.40±0.3081.60±0.0049.58±0.5157.68±0.1568.93±0.052.00
PPL53.61±0.1082.67±0.1174.54±0.1868.50±0.1368.14±0.2942.29±0.0778.80±0.0879.00±0.6943.72±0.8057.48±0.2964.87±0.103.40
Attn-Inf51.57±0.0468.80±0.0264.39±0.2664.84±0.1355.29±0.6043.21±0.0278.78±0.2065.40±1.0441.88±0.9756.38±0.2159.05±0.134.70
GRAFT62.63±0.0591.67±0.0481.11±0.2672.07±0.0870.48±0.0643.60±0.0479.63±0.1481.13±0.8358.18±0.7859.83±0.0570.03±0.061.40
MethodMMLUARC-EARC-CCSQAQASCMMLU-ProPIQAOBQAAGIEvalHellaMacroAvg.Rank↓Δ Base
Base69.02±0.0992.76±0.0888.65±0.0979.42±0.1779.27±0.3956.13±0.0983.22±0.1187.93±0.5073.52±0.8777.19±0.2078.71±0.104.50—
Random75.35±0.0196.93±0.0492.58±0.1580.51±0.0081.25±0.1659.95±0.0886.42±0.3089.13±0.3176.74±0.2769.48±0.1680.83±0.002.90+2.12±0.11
DCLM72.93±0.0797.14±0.0492.63±0.1078.95±0.0080.31±0.0663.30±0.0588.05±0.0889.00±0.2078.59±0.4068.89±0.1680.98±0.053.10+2.27±0.06
QuRating76.16±0.0796.63±0.0491.64±0.0979.69±0.0883.01±0.1664.29±0.0786.76±0.1390.53±0.7676.48±0.8069.13±0.4681.43±0.162.90+2.72±0.07
GRAFT76.47±0.0997.28±0.0692.15±0.0980.32±0.1383.12±0.0664.47±0.0986.80±0.1390.13±1.2179.48±0.5174.51±0.1682.47±0.101.60+3.76±0.19
  • W2c/Q4: efficient discussion

A: Thank you for your question. We will provide a more detailed efficiency analysis and clarify that the reported timings cover GPU computation. The top-K token analysis is performed once on the CPU using a sample of 10,000 documents and takes 0.234 seconds. The resulting anchor set is reused across the entire dataset through masking during the forward pass. This one-time preprocessing cost is therefore negligible when amortized over a large dataset. GRAFT benefits from avoiding vocabulary-wide loss computation, particularly when the tokenizer’s vocabulary is large. During the forward pass, the input consists of a token sequence and a corresponding mask identifying the anchor set, while the model retains its standard causal attention mechanism. In our experiments, we use batch sizes of 356, 32, and 12 for documents containing fewer than 500 tokens, 500–2,000 tokens, and more than 2,000 tokens, respectively.

Below, we provide a breakdown of the main computational components involved in calculating BF and PPL scores for each document.

Document length(tokens)Full forward pass and activation extractionGraph constructionBF reduction
<5005.4040.2390.214
500–200037.6710.8700.192
>2000131.8082.4960.228
Document length(tokens)Backbone forward passLM head,cross-entropy and loss reduction
<5005.4510.213
500–200036.9592.683
>2000128.8468.386

Therefore, graph construction and score aggregation require less computation than cross-entropy loss calculation. Loss-based methods operate over the full tokenizer vocabulary, which can incur substantial computational costs for backbones with large vocabularies.

  • Q6 Fix the images:

A: Thanks for your carefully reading. We've updated the iamges.

Reviewer_Bzn1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GRAFT is an unsupervised data-selection method for continual pretraining (CPT). It uses the model’s internal concept connectivity to find documents containing knowledge that has not yet been fully integrated. GRAFT builds a graph from similarities between anchor-token MLP activations at each layer and summarizes its connectivity with Branch Fraction (BF). Documents with low BF in only a few layers are selected for CPT.

The authors evaluate GRAFT on SmolLM2-1.7B, Qwen2.5-3B, and Qwen3-8B-Base against Random, DCLM, QuRating, PPL, and AttentionInfluence. GRAFT achieves the best average rank across all three models. In a mixed Math+STEM setting, PPL selects about 95% math data and degrades performance, while GRAFT maintains a roughly 55:45 balance. Additional experiments examine scoring throughput, layer-wise selection patterns, and connectivity changes under learning and forgetting data.

Critical strengths

1. Avoids domain skew without a target or domain labels. In the mixed Math and STEM experiment, PPL selects about 95% of its data from math and substantially hurts performance in both domains. GRAFT maintains a roughly 55 to 45 split and stable performance through 20B tokens without a reference model or domain labels. This provides strong evidence that BF captures more than predictability.

2. Performs consistently across models at low scoring cost. GRAFT achieves the best average rank on SmolLM2-1.7B, Qwen2.5-3B, and Qwen3-8B. It scores 7.61 documents per second, close to 7.40 for PPL and about twice as fast as AttentionInfluence. This makes GRAFT practical for CPT data selection.

Critical weaknesses

1. Not convinced the gains come from "knowledge connectivity" rather than from other factors.

(a) The improvements may come from answer formatting, not new knowledge. Evaluation is 0-shot CoT exact match on base models, and those models often score below chance (SmolLM2-1.7B gets 8.28 on 4-way HellaSwag). The candidate pool includes QA and instruction data, so even random selection raises HellaSwag to 35.09. A selector that simply picks more QA-style text would look strong under this protocol.

(b) The margins are about one point, from single runs. GRAFT beats the best baseline by 0.92 macro points on Qwen2.5-3B and 1.08 on Qwen3-8B, with no seed variance. The same base checkpoint already differs by 0.23 points between Table 1 and Table 3, so I can't tell how much of the gap is noise.

(c) Figure 7 does not support the selection rule actually used. In the BF < 0.1 row, accuracy goes up with more voting layers (59.4 at 1-3 layers, 62.6 at 15-18), and the best cells are in other rows. What the figure does show is that documents unfamiliar to many layers hurt. That is a weaker claim than "growing missing knowledge."

2. The paper describes a somewhat different method from the one it runs, and it can't be reproduced as written.

(a) The text and the experiments disagree in several places. The abstract says "middle layers," but the algorithm counts votes over all layers. The 10%->90% layer schedule in Sec. 2.2 never appears in the experiments. "Votes from layers 1-3" in Sec. 4 reads as layer indices, but it seems to mean vote counts.

(b) The speed claim is overstated. Line 268 says MLPs are computed only for anchor tokens, and I don't see how that works in a causal Transformer without changing the activations being measured. Figure 4 shows GRAFT at roughly PPL speed (slower on short documents), so it is faster only than AttentionInfluence.

(c) Key details are missing. These include how predecessor statistics and nodes are computed, the (K, τ) values per model, budget matching, and the training hyperparameters (see Questions).

Questions for authors

The following clarifications could change my assessment, in order of importance.

  1. Does GRAFT still outperform under an evaluation less sensitive to answer format? Please provide the 0-shot CoT prompt, answer extraction rule, and parse-failure rates. It would also be helpful to report likelihood-based or few-shot accuracy and how much QA or instruction-formatted data each selector selects. This would address my main concern in Weakness 1.

  2. Are the main gains statistically reliable? Please report results across seeds or confidence intervals for Tables 1 and 2.

  3. How was the final BF selection rule chosen? Figure 7 appears to favor settings other than BF below 0.1 with 1-4 voting layers. Please clarify how this setting was selected, whether the same rule is used across models, and report K and τ for each model. Please also define BFv2.

  4. Does GRAFT scoring require a full forward pass? Please clarify what is included in the runtime reported in Figure 4, including the model, batch size, predecessor-diversity lookup, and graph construction.

  5. How are selection budgets matched across methods? Please explain how GRAFT is matched to the baseline budget when its selected region contains a different number of tokens, and clarify whether PPL selects high or low perplexity data.

  6. Why do the Qwen2.5-3B Base scores differ between Tables 1 and 3?

  7. Could you clarify the Section 4 analysis? Do vote1to3 and vote29to32 refer to vote counts or layer indices? Is the learning-rate schedule restarted in the second stage of Table 5? For Figure 12, how are added and lost edges compared across different free-form responses, and how is response length controlled?

Suggestions

These are not required for my assessment.

  • Compare against RHO-1 and FineWeb-Edu, which would provide stronger baselines closer to the proposed selection setting.
  • Validate BF on a held-out split rather than on the same examples used for training.

Minor Issues

  • The AI disclosure contains contradictory template text about surveys, interviews, and transcription.
  • The metric called "Total Variation Entropy" in Appendix C.2 is a distance from uniformity, with lower values indicating greater uniformity.
  • In Table 2, OpenBook 90.00 is bolded while the higher 90.40 is underlined.
  • Tables 3 and 4 use proportions while the other tables use percentages.
  • Several references do not include a venue.
  • There are several typos: "Treatmetn", "phase" instead of "phrase", "BRACH FRACTION", "knowledege", and "PreTraining".
5 comments
  • W1a/Q1a. selected data formulation analysis and why for new knowledge.

A: Thanks for your question. Here we give the further analysis on how different methods select the data to train. We analysis it on Qwen2.5-3B based backbone model and formulate it into a. formulation b. document length c. token efficient for domain knowledge.

a. As shown in table, we report the data formulation that labeled by Nemtron-Specialized data. The results suggest GRAFT select the similar sampling weight camparing with the random setting, but others will select more QA formulation. This suggests the data formulation is a minor impact to the final result.

MethodFact-SeekingMultiple-Choice(QA)GenerativeMoral-Scenarios
Random82.11%16.24%1.61%0.036%
DCLM88.76%10.61%0.63%0%
QuRating41.98%52.18%5.67%0.169%
PPL62.64%30.79%6.56%0.010%
Attn-Inf43.37%50.47%6.01%0.144%
GRAFT79.28%19.51%1.12%0.103%

b. We further report the length of selected documents in the below table. The results show that GRAFT do not have the bias to select data that have more length or short length comparing with others.

MetricRandomDCLMQuRatingLow PPLAttn-InfGRAFT
Median length(tokens)563686.5102122.572

c. We further analysis the domain data distribution and token efficient for each domain. Specifically, we use Qwen3.5-9B to label all of the 10 benchmark with 18 various domain labels, and also for the selected data for each method.

In this way, we got the proportion of domain data selected by various methods and how the selected help increasing accuracy by token efficient \Delta_{acc}/{#token} We report the main results are shown in below table.

Fine-Grained DomainRandomDCLMQuRatingLow PPLAttn-InfGRAFT
Law3.95%/+12.742.42%/+12.923.25%/+18.332.35%/+20.282.03%/+25.614.06%/+13.56
Philosophy,Ethics&Religion0.48%/+143.660.45%/+30.011.66%/+39.310.71%/+64.450.80%/+55.010.50%/+164.03
Engineering&Manufacturing1.29%/+40.380.81%/+12.911.50%/+62.810.60%/+131.510.63%/+84.931.61%/+48.50
Chemistry1.17%/+36.191.07%/+1.653.72%/+24.253.59%/+13.623.13%/+5.130.93%/+80.00
Economics&Finance2.53%/+13.472.88%/+4.531.62%/+31.332.83%/+13.112.12%/+13.421.98%/+26.42
Business&Management0.45%/+53.720.29%/+35.000.60%/+76.450.15%/+205.420.24%/+144.930.43%/+99.75
Physics&Astronomy2.35%/+18.841.70%/+6.965.45%/+14.004.78%/+12.264.08%/+6.482.47%/+29.71
History&Archaeology33.79%/+1.1931.26%/−0.2013.08%/+3.9928.59%/+0.9616.59%/+1.9535.34%/+1.50
Medicine&Health1.80%/+21.381.30%/+4.764.86%/+15.282.57%/+27.792.45%/+11.672.04%/+35.95
Mathematics&Statistics8.20%/+1.417.88%/−0.3010.00%/+4.0517.33%/+0.5313.98%/+0.027.94%/+7.81
Social&Everyday Situations7.69%/+4.944.18%/+4.9411.17%/+4.202.26%/+16.248.07%/+4.907.47%/+5.91
Psychology0.29%/+143.720.19%/+43.561.20%/+42.350.28%/+142.690.42%/+74.990.35%/+139.79
Earth&Environment2.21%/+28.564.01%/+3.602.81%/+30.783.13%/+24.402.05%/+21.222.23%/+39.62
Objects&Space9.02%/+4.3314.24%/+1.043.84%/+12.866.67%/+6.946.88%/+5.446.26%/+8.67
Language&Literature14.70%/+3.3512.61%/−0.8619.97%/+3.3710.71%/+3.6324.38%/+1.2916.92%/+3.56
Biology&Ecology2.72%/+14.695.37%/+0.795.92%/+11.202.86%/+18.732.69%/+5.762.81%/+23.74
Tools&Procedures0.61%/+47.470.12%/+163.342.66%/+18.400.50%/+96.281.73%/+27.880.76%/+62.03
Computing&Programming3.14%/+11.024.93%/+0.886.67%/+10.906.06%/+9.305.80%/+3.982.40%/+25.71

The results suggest that all of method will chooce various of domains, but GRAFT will choose the most efficient tokens to help the model learn the specific tasks. GRAFT take the first place in token effient in 8 domains and 4 in second. This suggest that GRAFT is able to equally select the most efficient data for as much domains as possible. We will add the analysis details in Appendix F.

AuthorsOpenReview ↗

W1a/Q1/W1b/Q2/Q6

  • W1a/Q1: provide the 0-shot CoT prompt, answer extraction rule, and parse-failure rates.

A: Thanks for your comments. The 0-shot CoT prompt is: "Think through the problem step by step before answering. End your response with exactly two final lines: first "Answer: X", where X is the letter of the best choice, and then "END_OF_ANSWER". Reasoning:" In extraction step, we first detect "END_OF_ANSWER", and then detect the "Answer: X" to be the selection. Under this rule, we report the failures in the table below:

MethodMMLUARC-EARC-CCSQAQASCMMLU-ProPIQAOpenBookQAAGIEval SATHellaSwagOverall InvalidAverage
Base17.3916.1815.7018.6718.6510.979.2910.139.236.0713.4513.23
Random2.250.040.090.000.116.390.130.403.320.372.851.31
DCLM2.620.310.850.080.226.830.051.403.060.634.531.61
QuRating1.270.040.170.000.072.420.050.200.690.071.210.50
Low PPL1.990.040.340.000.002.490.000.202.370.131.760.76
Attn-Inf1.400.040.000.000.003.850.200.401.580.121.780.76
GRAFT1.730.080.110.080.112.400.161.071.110.131.380.70

In general, continual pretraining can improve answer accuracy. Both GRAFT and the baseline methods benefit from the thinking and answer extraction templates.

  • Q1: It would also be helpful to report likelihood-based or few-shot accuracy.

A: Thanks for the comments. Here we show the results of likelihood-based and few-shot accuracy on Qwen2.5-3B model.

Evaluation protocolBaseRandomDCLMQuRatingLow PPLAttn-InfGRAFT
0-shot likelihood56.13±0.0051.53±0.0252.54±0.0049.91±0.0150.20±0.0050.62±0.0152.17±0.00
5-shot73.31±0.0469.51±0.0265.44±0.0372.18±0.1171.69±0.0272.27±0.0670.85±0.03

Without thinking mode, all of methods cannot beat the base one. The corpus (Nemtron-Specialized-v1.2) contains knowledge-bearing and explanatory text. CPT optimizes modeling of this textual distribution, but does not directly constrain the probability ranking of answer options under benchmark QA prompts. Consequently, gains observed under generative evaluation need not translate into improved likelihood-based accuracy.

  • W1b/Q2/Q6: random seed.

A: Thanks for the comments. We've updated the random seed (3 runs) in Table 1 and Table 2 to show the variance of each methods and we've updated them in the paper.

MethodMMLUARC-EARC-CCSQAQASCMMLU-ProPIQAOpenBookAGIEvalHellaMacroAvg.Rank
Base43.93±0.0469.16±0.1557.85±0.0948.46±0.0944.85±0.1632.27±0.0967.30±0.1961.13±1.5557.38±0.2457.66±0.2854.00±0.125.70
Random55.17±0.0580.51±0.1569.48±0.1365.93±0.0860.01±0.4141.58±0.1276.24±0.1671.60±1.0044.88±0.9753.80±0.3761.92±0.074.30
DCLM43.81±0.0264.73±0.1158.87±0.0961.37±0.0553.42±0.0638.84±0.1270.89±0.2459.40±1.7439.24±0.1652.49±0.2954.31±0.156.50
QuRating62.09±0.0792.44±0.0981.80±0.1871.09±0.1471.89±0.1242.75±0.0778.40±0.3081.60±0.0049.58±0.5157.68±0.1568.93±0.052.00
PPL53.61±0.1082.67±0.1174.54±0.1868.50±0.1368.14±0.2942.29±0.0778.80±0.0879.00±0.6943.72±0.8057.48±0.2964.87±0.103.40
Attn-Inf51.57±0.0468.80±0.0264.39±0.2664.84±0.1355.29±0.6043.21±0.0278.78±0.2065.40±1.0441.88±0.9756.38±0.2159.05±0.134.70
GRAFT62.63±0.0591.67±0.0481.11±0.2672.07±0.0870.48±0.0643.60±0.0479.63±0.1481.13±0.8358.18±0.7859.83±0.0570.03±0.061.40
MethodMMLUARC-EARC-CCSQAQASCMMLU-ProPIQAOBQAAGIEvalHellaMacroAvg.Rank↓Δ Base
Base69.02±0.0992.76±0.0888.65±0.0979.42±0.1779.27±0.3956.13±0.0983.22±0.1187.93±0.5073.52±0.8777.19±0.2078.71±0.104.50—
Random75.35±0.0196.93±0.0492.58±0.1580.51±0.0081.25±0.1659.95±0.0886.42±0.3089.13±0.3176.74±0.2769.48±0.1680.83±0.002.90+2.12±0.11
DCLM72.93±0.0797.14±0.0492.63±0.1078.95±0.0080.31±0.0663.30±0.0588.05±0.0889.00±0.2078.59±0.4068.89±0.1680.98±0.053.10+2.27±0.06
QuRating76.16±0.0796.63±0.0491.64±0.0979.69±0.0883.01±0.1664.29±0.0786.76±0.1390.53±0.7676.48±0.8069.13±0.4681.43±0.162.90+2.72±0.07
GRAFT76.47±0.0997.28±0.0692.15±0.0980.32±0.1383.12±0.0664.47±0.0986.80±0.1390.13±1.2179.48±0.5174.51±0.1682.47±0.101.60+3.76±0.19
AuthorsOpenReview ↗

W1c/Q3/W2b/Q4/W2a/W2c

  • W1c/Q3: the selection rule actually used in experiments

A: Thank you for your question. We select the layers by layer voting numbers, instead of index. In practice, we start with a small number of voting layers (e.g., 1–3 in our main experiments) to avoid sharp performance drops near the boundary. As shown in Figure 7, accuracy drops abruptly from 55.7% to 48.3% when the number of voting layers exceeds 25. Outside this region, the selected data remain useful for training.

For example, to select 3B of data, we expanded the voting-layer count range from 15–18 to 7–18 because the 15–18 range yielded only 1.2B of data. The expanded range achieves an average accuracy of 62%, comparable to the result with 1–3 voting layers.

settingMMLUARC-EARC-CCSQAQASCMMLU-ProPIQAOpenBookQAAGIEval SATHellaSwagMacro
Vote1–362.63±0.0591.67±0.0481.11±0.2672.07±0.0870.48±0.0643.60±0.0479.63±0.1481.13±0.8358.18±0.7859.83±0.0570.03±0.06
Vote7–1861.16±0.0691.13±0.0680.32±0.2573.96±0.0869.98±0.1942.56±0.0279.52±0.2574.00±2.2555.22±0.7361.12±0.1168.90±0.31

The experiment in Figure 7 uses only 0.1B of injected data, so its results cannot be directly extrapolated to the 3B injection setting. Nevertheless, it reveals a trend that helps identify and avoid sharp performance drops. We therefore recommend running this small-scale experiment before large-scale training, while using a small number of voting layers is a reasonable starting point.

  • W2b/Q4: Computing time for GRAFT

A: Thank you for your question. We will revise the text to avoid overstating the speed advantage and provide a more detailed efficiency analysis. GRAFT benefits from avoiding vocabulary-wide loss computation, particularly when the tokenizer’s vocabulary is large. During the forward pass, the input consists of a token sequence and a corresponding mask identifying the anchor set, while the model retains its standard causal attention mechanism. In our experiments, we use batch sizes of 356, 32, and 12 for documents containing fewer than 500 tokens, 500–2,000 tokens, and more than 2,000 tokens, respectively.

Below, we provide a breakdown of the main computational components involved in calculating BF and PPL scores for each document.

Document length(tokens)Full forward pass and activation extractionGraph constructionBF reduction
<5005.4040.2390.214
500–200037.6710.8700.192
>2000131.8082.4960.228
Document length(tokens)Backbone forward passLM head,cross-entropy and loss reduction
<5005.4510.213
500–200036.9592.683
>2000128.8468.386

Therefore, graph construction and score aggregation require less computation than cross-entropy loss calculation. Loss-based methods operate over the full tokenizer vocabulary, which can incur substantial computational costs for backbones with large vocabularies. The algorithm details are shown in appendix A.3.

  • W2a: The text and the experiments disagree.

A: Thank you for the careful reading. We will revise the text to clarify two points. First, our observation that middle layers contribute most to identifying valuable data comes from the post hoc analysis in Section 4. Second, the percentages (10%–90%) specify the number of layers used for voting in our data selection experiments. For example, in Section 3.4, we group the number of voting layers into ranges of 1–3, 4–7, and so on, based on the 36 layers of Qwen2.5-3B. Thus, 1–3 voting layers correspond approximately to 0%–10% of the total layer count. We will revise the illustrations in Section 4 to explicitly label the number of voting layers (e.g., “1–3 voting layers”) and ensure consistent notation throughout the paper.

  • W2c: how predecessor statistics and nodes are computed, the (K, τ) values per model, and the training hyperparameters

A: The predecessor is computed by 10,000 random document. For all of experiments, we set (K=30, τ=0.7) for each setting. The training parameters are shown below. We've added more details in Section G.

HyperparameterQwen2.5-3B/SmolLM2-1.7BQwen3-8B-Base(1B-token runs)
Sequence length2,048tokens2,048tokens
Global batch size256sequences512sequences
OptimizerAdamWAdamW
AdamW$(\beta_1,\beta_2)$(0.9,0.95)(0.9,0.95)(0.9,0.95)\
Weight decay0.10.10.10.1
Peak learning rate1×10−51\times10^{-5}5×10−65\times10^{-6}
Learning rate scheduleCosine decayCosine decay
Warmup ratio0.5%0.5%
Minimum learning rate1×10−61\times10^{-6}5×10−75\times10^{-7}

Further, we provide the top 30 token that we used in the experiments for Qwen2.5-3B. The full is in appendix G.

.\n|?\n|in|,|\n|of|"|for|to|from|(|'s|at|on|and|during|with|that|by|or|after|,"|a|i|.\n\n|as|ing|play|."|de|

Reviewer_7aeiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GRAFT, a data-selection method for continual pretraining (CPT) that scores candidate documents by the connectivity of latent semantic concepts inside the target LLM. Experiments cover SmolLM2-1.7B, Qwen2.5-3B, and Qwen3-8B on ten knowledge/reasoning benchmarks, plus mixed STEM/math selection, efficiency comparisons, layer analysis, and catastrophic-forgetting analysis.

Critical strengths

1.Moving beyond token-level loss/perplexity to concept-connectivity in hidden MLP up-projections is a fresh angle. 2.GRAFT is evaluated on three model scales and ten benchmarks.

Critical weaknesses
  1. BF may select low-quality/ungrammatical documents rather than knowledge gaps. 2.The BF score is non-unique and only loosely validated as “knowledge familiarity.”
  2. Layer-voting procedure is unclear and possibly inconsistent.
  3. The BF score has two components: non-isolated coverage and degree structure. The paper should ablate: non-isolated coverage only, degree term only, random anchor tokens, shuffled edges, and BF computed at a single fixed layer vs. voting.
Questions for authors
  1. How does GRAFT avoid selecting low-quality/noisy documents whose low BF is due to incoherence rather than novelty? 2.What is the per-document correlation between BF and PPL, AttnInfluence, and actual CPT loss reduction? Is low BF predictive of downstream gain after controlling for document quality?
6 comments
AuthorsOpenReview ↗

W1/Q1/W2/W3

  • W1/Q1 low quality/ungrammatical documents distinguish:

A: Thank you for raising this concern. We apply document quality controls before constructing the candidate pool. At this stage, rule-based filters can screen for obvious text defects. In experiment, we already used the filtered data, where the candidate pool is high quality Nemtron-Specialized-v1.2. We also give how BF responds to token repetition, a specific form of noise. We randomly sampled 10,000 documents for which 1–3 layers satisfied the BF<0.1\mathrm{BF}<0.1 criterion. For each document, we constructed two variants of equal token length. In the repetition condition, we repeated a 5–10-token span from the original document. In the control condition, we added the same number of tokens while preserving their original order. For each condition, we averaged BF across the 10,000 documents at each of the 36 layers and counted how many layer means fell into each interval.

Mean BF per layeroriginal orderRepeat a 5–10 token span
<0.250
[0.2, 0.4)1511
[0.4, 0.6)516
≥0.6119

Under the tested perturbation, all five layers with a mean BF below 0.2 in the control reached at least 0.2 after repetition. The number of layers with mean BF in [0.4,0.6)[0.4,0.6) also increased from 5 to 16. This shows that BF is able to distinguish the repeated spans.

  • W2/W3: The BF score is non-unique/the BF score has two components: non-isolated coverage and degree structure. The paper should ablate: non-isolated coverage only, degree term only, random anchor tokens, shuffled edges

A: Thank you for raising this distinction. BF’s many-to-one mapping is a deliberate design choice. By combining non-isolated node coverage and degree structure, BF places diverse sparse connectivity patterns on a common scale for data selection. Appendix A.2 and Figure 15 illustrate this flexibility with structurally distinct graphs that receive the same BF score.

Further, We evaluated the BF components and anchor selection on the same 10,000 documents using Qwen2.5-3B Base and using threshold of τ=0.70\tau=0.70. We computed both metrics separately at each layer and averaged them equally across all 36 layers. H50=(1−∑bpb2)/(1−1/50)H_{50}=(1-\sum_b p_b^2)/(1-1/50) measures normalized score diversity, where pbp_b is the fraction of documents in bin bb and bin boundaries are defined by pooled score quantiles. The exact tie rate is the fraction of distinct document pairs with identical scores, including pairs where both scores are zero.

ScoreMean H50 ↑Exact tie rate ↓
Non-isolated0.803919.12%
Degree0.807318.16%
BF0.815016.80%

Under the same graph construction, combining the two terms yields the highest score diversity and reduces exact ties by 1.36 percentage points relative to the degree term, the stronger of the two individual components.

Anchor coverage helps explain the difference in score resolution. The predecessor-based method found at least one anchor in every document in this sample. The random token types left 4,048 documents without anchors, and assigning zero BF to those documents contributed to the higher tie rate.

Anchor selectionDocuments with no anchorsMean H50 ↑Exact tie rate ↓
Predecessor Top300 (0.00%)0.815016.80%
Random 304,048 (40.48%)0.116187.94%

In GRAFT, the edge set is determined by the anchor representations. For a given document, model checkpoint, and layer, each selected anchor has a fixed representation. We connect an anchor pair when its cosine similarity exceeds τ\tau and orient the edge according to token order. Thus, fixing the anchors and τ\tau uniquely determines the edge set. Shuffling edges with the anchors held fixed would replace the cosine-based connection rule and constitute a separate graph-construction baseline. We will clarify this dependency in the method description.

  • W2: BF only loosely validated as “knowledge familiarity.”

A: Thanks for your question. We would like to clearify the "knowledge familiarity" from to aspects.

First, we've discussed in Section 4. The knowledge familiarity is reflected in increasing the connections between question and reasoning connections and reasoning itself connections.

Second, we will add another domain analysis in Appendix F.1. Specifically, we use Qwen3.5-9B to label all of the 10 benchmark with 18 various domain labels, and also for the selected data for each method. In this way, we got the proportion of domain data selected by various methods and how the selected help increasing accuracy by token efficient \Delta_{acc}/{# token} We report the main results are shown in below table.

Fine-Grained DomainRandomDCLMQuRatingLow PPLAttn-InfGRAFT
Law3.95% / +12.742.42% / +12.923.25% / +18.332.35% / +20.282.03% / +25.614.06% / +13.56
Philosophy, Ethics & Religion0.48% / +143.660.45% / +30.011.66% / +39.310.71% / +64.450.80% / +55.010.50% / +164.03
Engineering & Manufacturing1.29% / +40.380.81% / +12.911.50% / +62.810.60% / +131.510.63% / +84.931.61% / +48.50
Chemistry1.17% / +36.191.07% / +1.653.72% / +24.253.59% / +13.623.13% / +5.130.93% / +80.00
Economics & Finance2.53% / +13.472.88% / +4.531.62% / +31.332.83% / +13.112.12% / +13.421.98% / +26.42
Business & Management0.45% / +53.720.29% / +35.000.60% / +76.450.15% / +205.420.24% / +144.930.43% / +99.75
Physics & Astronomy2.35% / +18.841.70% / +6.965.45% / +14.004.78% / +12.264.08% / +6.482.47% / +29.71
History & Archaeology33.79% / +1.1931.26% / −0.2013.08% / +3.9928.59% / +0.9616.59% / +1.9535.34% / +1.50
Medicine & Health1.80% / +21.381.30% / +4.764.86% / +15.282.57% / +27.792.45% / +11.672.04% / +35.95
Mathematics & Statistics8.20% / +1.417.88% / −0.3010.00% / +4.0517.33% / +0.5313.98% / +0.027.94% / +7.81
Social & Everyday Situations7.69% / +4.944.18% / +4.9411.17% / +4.202.26% / +16.248.07% / +4.907.47% / +5.91
Psychology0.29% / +143.720.19% / +43.561.20% / +42.350.28% / +142.690.42% / +74.990.35% / +139.79
Earth & Environment2.21% / +28.564.01% / +3.602.81% / +30.783.13% / +24.402.05% / +21.222.23% / +39.62
Objects & Space9.02% / +4.3314.24% / +1.043.84% / +12.866.67% / +6.946.88% / +5.446.26% / +8.67
Language & Literature14.70% / +3.3512.61% / −0.8619.97% / +3.3710.71% / +3.6324.38% / +1.2916.92% / +3.56
Biology & Ecology2.72% / +14.695.37% / +0.795.92% / +11.202.86% / +18.732.69% / +5.762.81% / +23.74
Tools & Procedures0.61% / +47.470.12% / +163.342.66% / +18.400.50% / +96.281.73% / +27.880.76% / +62.03
Computing & Programming3.14% / +11.024.93% / +0.886.67% / +10.906.06% / +9.305.80% / +3.982.40% / +25.71

The results suggest that all of method will chooce various of domains, but GRAFT will choose the most efficient tokens to help the model learn the specific tasks. GRAFT take the first place in token effient in 8 domains and 4 in second. This suggest that GRAFT is able to equally select the most efficient data for as much domains as possible.

Third, we will add another held-out experiment to analysis the knowledge familarity in Appendix F.2. We conducted an additional analysis on held-out data to examine domain effects. We grouped the subdomains into two broad categories: STEM and humanities (history, philosophy, psychology, economics, business, law, and language). For each category, we selected 0.3B tokens for continual pretraining (CPT). We then measured layer-wise changes in BF scores relative to the base model on held-out documents from both categories. The table below reports the top five layers with the largest changes. Here, “A CPT × B” denotes continual pretraining on domain A and evaluation on domain B.

LayerHumanities CPT × HumanitiesHumanities CPT × STEMSTEM CPT × HumanitiesSTEM CPT × STEM
L20+90.44%+81.60%+75.50%+84.21%
L21+77.63%+70.20%+67.38%+68.10%
L22+55.52%+51.69%+46.46%+48.49%
L24+50.98%+43.69%+46.05%+43.95%
L23+46.10%+41.47%+41.13%+42.00%
Top-5 mean · All documents with valid scores+64.13%+57.73%+55.31%+57.35%

Across these top five layers, the average increase in BF scores is larger when the evaluation domain matches the CPT domain, suggesting the model is learning knowledge.

AuthorsOpenReview ↗

W3/W4/Q2/Q3

  • W3: Layer-voting procedure is unclear and possibly inconsistent.

A: Thank you for pointing out this ambiguity. In our implementation, all Transformer layers participate in scoring each document. A layer contributes one vote when its BF score satisfies the specified criterion, and a document is retained when the total vote count falls within a prescribed range. Selection is performed once before CPT; there is no progressive expansion of participating layers or voting schedule during training. The phrase “votes from layers 1–3” was misleading: it refers to a vote count of 1–3, not layer indices 1–3. We will correct this wording throughout, replace the description suggesting progressive layer expansion in Appendix C.1.

  • W4: BF computed at a single fixed layer vs. voting

A: Thank you for raising this question. As discussed in Section 4, votes tend to come from middle layers, particularly layers 18–23. Selecting data using BF from layer 20 alone yields a macro-average accuracy of 67.68%, within 2.35 percentage points of the 70.03% achieved with layer voting. Voting performs better in this comparison and allows different layers to contribute to the selection decision for different documents.

  • Q2: What is the per-document correlation between BF and PPL, AttnInfluence, and actual CPT loss reduction?

A: Thanks for your question, we will add these details in the Appendix. The documents selected by PPL, Attn-Inf and BF is totally different. In training stage, with the same 10k evaluation data, BF has the lowest Token-weighted NLL comparing with other method and lower perplexity value.

Method AMethod BShared DocumentsUnion DocumentsJaccard
PPLAttn-Inf2,436,65734,252,8217.11%
PPLBF1,503,68352,722,0522.85%
Attn-InfBF1,023,36350,163,6362.04%
ModelToken-weighted NLL ↓Perplexity ↓
PPL1.66055.2619
Attn-Inf1.53594.6456
BF1.51804.5629
  • Q3: Is low BF predictive of downstream gain after controlling for document quality?

A: Thanks for your valuable question. At comparable levels of document quality, we expect the predictive value of low BF for downstream gains to depend on both which layers exhibit low BF and how many do. Section 4 examines the effect of layer choice, and Figure 7 shows how downstream performance varies with the number of layers with low BF. As we analysis in Catastrophic Forgetting part in Section 4, when low BF is confined to a few layers, it may indicate a gap that the model can fill by building on stronger concept connections in other layers. At a matched 1B-token budget, training on the 1–3-vote subset yields 69.66%, while training on the 29–32-vote subset yields 48.07%, below the base model’s 54.01%. The latter model’s response graphs also lose more existing connections and gain fewer new ones (Figure 12), consistent with the observed forgetting. This is why we utilize layer voting method to select the suitable documents for models to learning.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your reviews and feedback. We would like to draw attention to several relevant details already included in the original submission. Below, we summarize these details alongside the additional evidence provided during the rebuttal:

  1. Comparability of data quality. Quality filtering is an upstream step shared by GRAFT and the comparison methods. Our work focuses on selecting data that improve models’ core capabilities from an already quality-filtered candidate pool. Accordingly, our original experiments used the high-quality Nemotron-Specialized-v1.2 dataset, with the same candidate data available to every method(Section 3.1, Line 242, Appendix B.1). The resulting distributions of selected data were analyzed post hoc; they were not prescribed in advance in the main experiments. Designing rule-based filters for low-quality, ungrammatical, or repetitive documents is beyond our scope. Nevertheless, during the rebuttal, we added experiments examining GRAFT’s robustness to lower-quality data.

  2. Experimental coverage and robustness. Our original submission evaluated the method across 17 benchmarks comprising 57,044 evaluation examples(Section 3.1, Appendix B, Table 1, Table 3, Figure 5, Figure 6): 44,781 for knowledge QA and 12,263 for mathematical QA. This coverage allows us to assess performance across a broad range of tasks. To further examine robustness, we added experiments with multiple random seeds for Qwen2.5-3B and Qwen3-8B during the rebuttal. GRAFT maintains the strongest overall performance in these comparisons, providing further evidence that its gains are consistent across seeds.

  3. Knowledge-gap interpretation. We use latent knowledge to refer to contextual semantic concepts and their connections within the model, drawing on the LLM interpretability literature discussed in our Related Work. BF provides an empirical proxy for this connectivity through associations among contextual semantic representations in intermediate layers. More important, the method gives a scalable method for interpretable data selection in continual pretraining. Our original submission examined how this topology relates to downstream reasoning in Section 4. During the rebuttal, we added experiments suggesting topological differences between representations of known and unknown data, providing further evidence for the proposed interpretation.

We remain open to further discussion and suggestions. If concerns remain unresolved, we would appreciate specific feedback on where our reasoning or evidence falls short and why, so that we can make targeted improvements to the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.