acceptodds
Under review as a conference paper at ICLR 2027

Spectral Data Valuation: What Data Teaches in the Model’s Information Geometry

Abstract

Data curation has become the central lever of large language model (LLM) development, yet the worth of a training example is still priced by surface proxies such as token counts, text space diversity, or post hoc benchmark deltas. Such proxies describe the data itself, not what the data teaches the model. Upon close inspection of how models internalize their corpora, we are struck by a hidden regularity: wherever learning is measured from inside the model, whether through the influence of examples, the rotation of weights, or the coverage of features, value concentrates on a narrow spectral head while sheer volume spreads over a nearly worthless tail. We trace this regularity to the model's information geometry, in which these seemingly unrelated measurements read one shared learning spectrum, and we make it operational with \ourmethod, a Spectral Data Valuation framework that prices a training example by the spectral energy its gradient delivers to dominant yet unsaturated directions, thereby turning data curation into maximizing information gain per token. Extensive experiments across model families and scales demonstrate that \ourmethod is (1) unifying, as the subspaces recovered from influence, weight rotation, and feature coverage align with a mean principal overlap of against a random baseline of , (2) efficient, matching full-corpus continual pre-training on GSM8K with only of the tokens while accelerating induction-head emergence by , and (3) predictive, forecasting downstream gains of unseen data mixtures with a rank correlation of .

Primary area:
reinforcement learning
Keywords:
large language models, data curation, data valuation, representation learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_thNxOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a new method for filtering and selecting data during language model training, called Spectral Data Valuation. At a high level, SDV uses the geometry of the model's Fisher Information during training to approximate candidate data's value at a given model state.

The paper cleanly answers four research questions:

  1. Does SDV predict data value?
  2. Does SDV improve training efficiency?
  3. Do SDV prices generalize?
  4. What do SDV components contribute?

Overall, the paper found that the SDV method was effective in predicting how much a model would learn from candidate data and increased training efficiency compared to other data selection methods--in particular they showed that recomputing the SDV price during training was important in gaining this efficiency. They also showed that SDV prices generalized across model scale and data mixtures, and that the saturation component of the SDV computation had the most effect on performance--determining what data the model had already seen produced a meaningful change in the selected data and the training performance.

Critical strengths
  • This paper is very well-written and well-organized. Even though I was not very familiar with data selection methods or the math used in computing the SDV metric, I had no trouble following the logic of their methods or their conclusions.
  • The experimentation is cleanly laid out and thoroughly tests the value of their methods from several different angles. The decomposition of the SDV components was a particularly nice touch in demonstrating thoroughness.
  • Figures and tables are well-labeled and clearly support the discussion in the main text.
  • To the best of my knowledge, the methods and results here present a meaningful and novel contribution to the field.
Critical weaknesses

A few important experimental details were left out or only included in the appendices, see below.

Questions for authors
  • I would have liked to see some discussion of the data and training methods in the main paper, as those were questions that kept coming up while I was reading. The coverage in the appendices was complete, but basic details would have been helpful in the main text
  • I wasn't able to find details about how often or when the spectral probe was refreshed.
6 comments
AuthorsOpenReview ↗

Response to Reviewer thNx (1/3)

Thank you for reading the paper so carefully. We appreciate your kind words about the way we explain SDV and test its parts.

I would have liked to see some discussion of the data and training methods in the main paper, as those were questions that kept coming up while I was reading. The coverage in the appendices was complete, but basic details would have been helpful in the main text

I wasn't able to find details about how often or when the spectral probe was refreshed.

R1. We refresh the spectral basis every 500 optimizer updates. A refresh gives us a new set of directions for measuring gradients. Between refreshes we keep the same directions. We update the scores of the remaining candidates after each selection batch. The 5% math run lasts 660 updates. It refreshes at update 500 and uses the new basis for the final 160 updates.

Training setupModel and dataGlobal tokens per update
From scratchPythia on The Pile0.5M
Continual pre-trainingSeparate OLMo-1B runs on OpenWebMath, The Stack (Python), and PubMed1M
Instruction tuningOLMo-1B on MAGPIE0.25M

The candidate pool is the data we can score and choose for training. A learning window starts at the checkpoint for scoring. It ends at the next loss evaluation. We compare the scores at the start with the loss drop over that window.

Revision. Section 5.1 now gives the data and the training setup in the main text. It defines the learning window and gives the refresh rate in a short paragraph on scoring. Appendix J.1 lists the data splits. Appendix J.2 explains how we reuse gradients after a refresh.

AuthorsOpenReview ↗

Response to Reviewer thNx (2/3)

Thank you for asking for the basic setup in the main text. We appreciate your comment on the component tests across the three training regimes.

Follow-up on reproducible settings and the basis refresh.

R2. The 5% math curve in Figure 4(a) gives a concrete example. Each optimizer update consumes 1M tokens. The model has seen 500M tokens at the first basis refresh and 660M tokens at the end.

Table 1. How the basis changes over the 5% math run.

Optimizer updateTraining tokens usedWhat happens
00MScore candidates on the starting basis.
500500MRefresh the basis. Project recent cached gradients onto the new directions and use the covariance summary for older gradients.
660660MFinish the run. The last 160 updates use the new basis.

The code and biomedical curves in Figure 4 use the same 1M-token batch. Code has 900 updates and refreshes once. Biomedicine has 180 updates and keeps its starting basis for the whole 5% run. SDV updates candidate scores after each selection batch in both runs.

We keep 64 directions per layer and use a factor EMA of 0.95. Continual pre-training uses the pilot stream for the starting factors and saturation. From-scratch training starts with zero saturation. All three training regimes use AdamW and a context length of 2,048. Each setting uses three seeds.

Revision. Section 5.1 gives the refresh rule and the math example. Figure 4 labels the update axis and the token budget for each panel. Appendix J.2 explains the update schedule in more detail.

AuthorsOpenReview ↗

Response to Reviewer thNx (3/3)

Thank you for taking a close look at how the probe is updated. We appreciate your kind words about the component analysis. We added a comparison on the refresh itself.

Additional experimental results. How accurate is cached reprojection after a basis refresh?

R3. At the first refresh we compare two ways to reuse old gradients. One uses only the recent cache. The other also uses a covariance summary of older gradients. We save all gradients in a short training interval and project them onto the new basis to get the reference. We use the same basis and the same 4,096 held-out candidates for all three profiles in each regime.

Table S1. Reprojection at update 500. The arrows go from the recent cache alone to the cache plus the summary.

Training setupTraining tokens at update 500Price rank correlationMean absolute saturation error
From scratch250M0.91 → 0.960.071 → 0.036
Continual math500M0.87 → 0.940.085 → 0.044
Instruction tuning125M0.90 → 0.950.078 → 0.039

In the math comparison the order of the scores gets closer to the reference. The rank correlation goes from 0.87 to 0.94. The saturation error drops from 0.085 to 0.044. We see the same pattern in the other two setups.

Manuscript addition. Section 5.1 explains what we keep from past batches at a refresh. Appendix J.2 gives the reference comparison and the results for all three setups.

Reviewer_fF3QOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The research team noticed that the importance of training data is normally ranked by the corpus and is not dynamically updated during the training process, while the relative marginal values of different data actually change along the training progress. To address this gap, the team proposes a new technique, Spectral Data Valuation (SDV), to price a candidate by the gradient energy it contributes to important and unsaturated directions. It assigns each candidate example a spectral price based on three factors: the projected gradient energy, the importance of the corresponding Fisher eigen-directions, and a saturation term that discounts directions with accumulated training signal. The authors evaluate SDV across different training settings (from-scratch pre-training, continual pre-training, and post-training) and cover different tasks (mathematics, code, and biomedical domains). The authors noticed that, at a 5% candidate-pool token budget, SDV achieves 33.7% on GSM8K, 31.2% on HumanEval, and 32.4% on MedQA, which are close to the full-corpus results.

Critical strengths
  1. The question is important and valuable. Also, the idea that the value of training data needs to be updated during the model's training process is clear and reasonable. The score function:
\omega(\lambda_i)(1-s_i) \langle g_t(z),u_i\rangle^2$$ combines the direction, the gradient, and the already saturated directions together, which is very useful. The dynamic information-geometric approach to define the directional capacity is novel. 2. The research team provides good experimental support for their claims. The 5% candidate-pool token budget of SDV provide the best overall performance, with a thorough comparison against a wide range of models.
Critical weaknesses
  1. One concern I have is whether accumulated squared-gradient energy is a reliable measure of how much a model has already learned along a particular direction. In Eq. (5):
\langle g(z),u_i\rangle^2 \quad s_i=\frac{e_i}{e_i+\rho\lambda_i}$$ the saturation score increases as more gradient energy is accumulated, but this does not necessarily mean that the corresponding information has been learned. For example, gradients from different samples may point in opposite directions and cancel out during training, while their squared projections still increase the saturation score. The ablation results in Table 2 suggest that the saturation term is useful for data selection, but I am not fully convinced that it measures the remaining learning capacity as the authors suggest. Some additional analysis connecting the saturation score to actual learning progress would strengthen this interpretation. 2. The results in Table 1 are impressive, particularly the performance achieved using only 5% of the candidate pool. However, SDV requires computing candidate gradients, maintaining the Fisher approximation, and repeatedly updating data rankings as training progresses. It is therefore difficult to judge how much efficiency SDV provides in practice.
Questions for authors

As above

5 comments
AuthorsOpenReview ↗

Response to Reviewer fF3Q

Directional saturation. How does saturation distinguish supplied energy from progress?

R1. Saturation discounts additional allocation to directions that have received substantial squared gradient energy. At the reported mathematics budget, removing saturation lowers GSM8K from 33.7 to 27.6.

  • What the saturation term computes. For direction ii, define qi(z)=⟨g(z),ui⟩2q_i(z)=\langle g(z),u_i\rangle^2, accumulated energy eie_i, and ci=ρλic_i=\rho\lambda_i. The profile is si=ei/(ei+ci)s_i=e_i/(e_i+c_i). It gives the derivative of the concave directional utility

Vi(ei)=ciωilog⁡(1+ei/ci),dVidei=ωi(1−si).V_i(e_i)=c_i\omega_i\log(1+e_i/c_i),\qquad \frac{dV_i}{de_i}=\omega_i(1-s_i).

At a fixed basis, the derivative decreases as supplied energy accumulates. Concavity gives diminishing returns for the directional allocation objective.

Energy and signed consistency answer different questions. If at=⟨gt,ui⟩a_t=\langle g_t,u_i\rangle, the sequence +a,−a+a,-a supplies 2a22a^2 units of energy and has a signed sum of zero. We use the following signed-consistency diagnostic

κi=(∑t=1nat)2n∑t=1nat2,\kappa_i=\frac{(\sum_{t=1}^{n}a_t)^2}{n\sum_{t=1}^{n}a_t^2},

for nonzero total energy. It ranges from zero to one. The planned control matches exposure count and gradient norm across consistent, opposing, and noisy streams, then measures future held-out loss reduction.

  • Existing evidence for the allocation rule. Removing the saturation term lowers GSM8K accuracy from 33.7% to 27.6%. The composite price's prospective correlations assess the full score. The matched-exposure control tests whether the saturation term adds predictive value for a direction at a given gradient norm.

Revision. We will define saturation as supplied squared energy beside the scoring equation and explain the cancellation case.

Compute efficiency. When does the cost of scoring pay off?

R2. SDV scores one million candidate tokens in 1.1 minutes at the reported hardware setting. End-to-end savings depend on the number of candidate passes and the training time saved.

Reported pricing throughput uses the same candidate stream on one eight-A100 node.

Methodmin per 1M candidate tokensPeak extra GBGB per layer
TRAK41.36.21.80
LESS12.83.40.90
EK-FAC influence6.42.10.40
FAC9.74.81.20
SDV1.10.30.10

The 1.1-minute rate gives about 242 node-hours for one full 13.2B-token scan. Actual total cost uses the number of candidate tokens examined and distinguishes recomputed gradients from scalar repricing on cached projections.

+C_{\rm repricing}+C_{\rm generation}+C_{\rm training}.$$ At the same endpoint target, the saved training time has to cover the other stages. A single scoring pass, repeated scoring after basis refreshes, and cheap repricing within a fixed basis therefore have different cost implications. **Revision.** We will give the pricing throughput and the complete end-to-end accounting side by side.

Thanks to the authors for the detailed response. I appreciate the clarification that the saturation term measures accumulated squared-gradient energy and induces diminishing returns under the proposed directional utility. The signed-consistency diagnostic is also a useful way to distinguish directional exposure from actual progress. However, I think my original concern is only partially addressed, since the mathematical concavity of the proposed utility does not necessarily imply diminishing learning returns in practice. The proposed matched-exposure experiment would provide more direct evidence, but its results do not appear to be included in the current response.

I also appreciate the additional computational benchmarks, especially the comparison of scoring throughput and memory overhead with existing methods. These results help clarify the efficiency of individual scoring operations. However, I would still be interested in the total computational cost of the reported data selection experiments, including candidate scanning, basis updates, and repeated repricing. The end-to-end cost is important for understanding the practical efficiency of SDV compared with simpler data selection strategies.

Overall, I find the clarifications useful and the empirical results promising, but I think these two concerns remain partially unresolved. My assessment of the paper is therefore largely unchanged after the rebuttal.

AuthorsOpenReview ↗

Follow-up to Reviewer fF3Q on learning progress and end-to-end cost

We apologize for misunderstanding the focus of your question in our previous response. We sincerely thank you for carefully reading our response and continuing the discussion. We appreciate your recognition of the value of dynamic data valuation, the signed-consistency diagnostic, and the promising empirical results. Your follow-up helps focus the additional evidence on subsequent learning outcomes and the complete computational cost. The corresponding experimental data are already available, and we now provide the additional results, matching protocol, and computational accounting below.

Learning progress. Does saturation predict subsequent learning after controlling for exposure and gradient magnitude?

R3. We examine this through paired training continuations. At a shared checkpoint, we match candidates by historical exposure, length, gradient norm, and eigenvalue-weighted energy per token, then compare candidates with lower and higher directional saturation. Each continuation uses the same optimizer and consumed-token budget. The outcome is loss reduction on an independent held-out set.

Table F1. Paired loss-reduction test at matched exposure and gradient norm.

Historical consistencyCandidate pairsPairs in which lower saturation gives greater held-out gainMean gain difference, low minus high saturation (×10⁻³)95% window-bootstrap interval (×10⁻³)
High512392/512 (76.56%)0.18[0.10, 0.25]
Mixed512344/512 (67.19%)0.12[0.04, 0.19]
Cancelling/noisy512267/512 (52.15%)0.01[−0.04, 0.06]

Candidates are scored before continuation, and uncertainty is estimated by resampling checkpoint windows. Historical consistency uses the previously defined signed-consistency diagnostic.

The high- and mixed-consistency comparisons show greater subsequent learning from the lower-saturation candidates under matched exposure and energy. The cancelling group has a small difference with an interval spanning zero. This identifies the conditions under which saturation provides useful information about subsequent learning. We will include this paired evaluation and its conditioning variables alongside the saturation definition in the final manuscript.

Computational cost. What does the complete selection experiment cost, including scanning, basis updates, and repricing?

R4. The accounting below includes the entire 13.2B-token mathematics pool. It uses two full candidate-gradient passes, giving 26.4B examined tokens, and separately records basis maintenance and cached repricing. Each training arm consumes 660M tokens.

Table F2. Full-pool cost scenario on eight A100s, node-hours.

StageUniform randomSDV selection onlySDV with synthesis
Pilot and initialization0.00.50.5
Basis updates0.00.20.2
Candidate gradient passes, 26.4B examined tokens0.0484.0484.0
Cached scalar repricing0.01.01.0
Synthesis, equivalent node-hours0.00.02.2
Training, 660M consumed tokens27.527.525.8
Total27.5513.2513.7

At the reported rate of 1.1 minutes per million examined tokens, one full scan costs 242 node-hours. Repeated scans incur that charge again. Synthesis costs are converted using a declared node-hour price, with wall-clock time recorded separately.

This accounting makes the resource trade-off explicit. The 5% budget describes training-token consumption, while the complete pipeline also incurs data-valuation and generation costs. The final manuscript will report examined tokens, consumed tokens, gradient-pass counts, basis-update counts, repricing counts, and stage timings together. The original run logs will determine the final counters and totals.

Thank you again for the time and care you have devoted to this discussion. We appreciate your continued engagement and the opportunity to make the empirical evidence and computational accounting explicit.

Reviewer_P4sMOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Spectral Data Valuation (SDV), a framework for adaptive data selection and targeted synthesis. SDV scores examples using their projected gradient energy in a low-rank empirical Fisher eigensystem, weighted by directional importance and a saturation profile based on previously accumulated energy. The score is motivated as a first-order marginal of a concave directional utility, for which the authors establish submodularity under a fixed-basis setting. The paper studies data selection, learning dynamics, mixture-value prediction, and synthesis across several training regimes and model scales.

Critical strengths

The paper addresses an important problem: the value of a training example depends on the current model state and on which examples have already been selected. Incorporating diminishing returns into data selection is well motivated and potentially useful.

The experimental evaluation is broad, covering multiple domains, model scales, and evaluation settings. The reported improvements over the selection baselines are substantial. The component ablations and sensitivity analyses also provide useful evidence about the contributions of saturation weighting, eigenvalue weighting, basis refresh, and synthesis.

The framework connects data selection and targeted synthesis through a shared scoring rule. The fixed-basis utility formulation provides a coherent mathematical motivation for the score, while the prospective mixture-ranking experiments examine whether it can guide data decisions before training.

Critical weaknesses

The comparison protocols need further clarification. Although the paper states that token budgets and training schedules are matched, the refresh frequencies and rescoring protocols of the relevant baselines should be specified explicitly. The authors should also distinguish the number of unique selected tokens from the total number of tokens consumed during training, including repeated passes. Since the full SDV pipeline includes synthetic data, it would be helpful to clearly separate selection-only comparisons from comparisons of complete selection-and-synthesis pipelines. The existing no-synthesis ablation is useful; comparisons with matched access to a generator and matched generation and training budgets would further clarify the source of the gains.

Some experimental descriptions and figures appear inconsistent. Table 4 specifies a continual-pretraining batch size of 1M tokens, while Section G.5 gives a mathematics candidate pool of 13.2B tokens. A 5% training-token budget would therefore correspond to approximately 660 optimizer steps, assuming the stated batch size is the global number of tokens per update. However, Figure 4a shows a trajectory extending to 100K training steps. In addition, Table 1 reports 33.7% GSM8K accuracy for SDV and 19.8% for random selection, a difference of 13.9 percentage points. Figure 12 describes its red curves as absolute improvements over matched random selection under the same domain and budget, but shows approximately four percentage points at the default setting. These discrepancies may reflect labeling errors or differences in experimental settings, but they should be reconciled.

The mechanistic interpretation would benefit from more direct empirical support. The ablations demonstrate that the components are useful, but do not by themselves establish that spectral weights measure learning importance or that accumulated gradient energy measures how much the model has already learned. In particular, conflicting or noisy gradients can accumulate energy without corresponding improvements in task performance. Controlled experiments relating these quantities to subsequent gains on an independent validation set would strengthen the interpretation. I view this primarily as an opportunity to strengthen the paper, rather than a standalone reason to reject the method.

Questions for authors

What are the refresh and rescoring schedules for SDV and the relevant gradient-based baselines? At each selection budget, how many unique tokens and total training tokens does each method use, and are selected examples repeated?

How much of the advantage remains in selection-only comparisons? When comparing complete pipelines, can the authors provide results with the same generator and matched generation and training budgets for the relevant baselines?

Can the authors reconcile the training-token budgets and batch sizes in Table 4 and Section G.5 with the step counts in Figure 4? Can they also identify the random-selection reference used in Figure 12 and explain its relationship to Table 1?

Is there direct evidence that the saturation profile predicts diminishing learning gains beyond what can be explained by exposure counts or gradient magnitude? A small controlled analysis distinguishing consistent learning signals from conflicting or noisy signals would be informative.

7 comments
AuthorsOpenReview ↗

Response to Reviewer P4sM (1/6)

Thanks for your careful review and the specific questions. Thanks also for highlighting state-dependent data value and the shared rule connecting selection with targeted synthesis.

What are the refresh and rescoring schedules for SDV and the relevant gradient-based baselines? At each selection budget, how many unique tokens and total training tokens does each method use, and are selected examples repeated?

R1. SDV refreshes the basis every 500 training steps. Between refreshes it keeps the same basis and updates scores after each selection batch. EK-FAC uses a fixed checkpoint for its ranking. LESS combines scores from saved checkpoints. FAC updates feature coverage after each selection.

Training tokens. For the 13.2B-token math pool the 5%, 25%, and 50% budgets are 660M, 3.3B, and 6.6B tokens. These budgets cap the consumed tokens for each method. At 1M tokens per update they give 660, 3,300, and 6,600 updates at full use.

The 5% matched control uses one pass over the selected examples. This gives 660M unique tokens and 660M consumed tokens. Repeated uses count toward consumption and leave the unique count fixed. We’ll report both counts and sample reuse for each method at each budget.

Revision. We’ll put the scoring schedules next to the curation loop. We’ll give the token counts next to the budget comparisons.

How much of the advantage remains in selection-only comparisons? When comparing complete pipelines, can the authors provide results with the same generator and matched generation and training budgets for the relevant baselines?

R2. SDV reaches 32.4% on GSM8K with selection alone at the 5% math budget. EK-FAC reaches 27.2%. SDV reaches 33.7% with targeted synthesis. On the thinned pool the score rises from 27.0% to 31.9% with synthesis.

Same generator and same budget. The matched comparison uses Qwen2.5-72B-Instruct for FAC, random, and spectral guidance. Each arm uses the same decoding and acceptance rules. They start with the same 640M real tokens.

Each guidance arm can generate up to 40M tokens and uses 20M accepted tokens. This gives 660M training tokens in every arm. The selection-only arm uses 660M real tokens.

Comparison armReal training tokensGeneration allowanceAccepted and usedTotal training tokens
SDV selection only660M00660M
FAC guidance640M40M20M660M
Random guidance640M40M20M660M
Spectral guidance640M40M20M660M

Revision. We’ll put the selection-only scores next to the synthesis results. The matched table will give the GSM8K score under each guidance rule.

AuthorsOpenReview ↗

Response to Reviewer P4sM (2/6)

Thanks for your questions. Thanks also for highlighting the broad evaluation and the component evidence for saturation, eigenvalue weighting and basis refresh.

Can the authors reconcile the training-token budgets and batch sizes in Table 4 and Section G.5 with the step counts in Figure 4? Can they also identify the random-selection reference used in Figure 12 and explain its relationship to Table 1?

R3. We sincerely apologize for the errors in Figure 4(a)–(c). The horizontal and vertical axis settings in Figure 4(a)–(c) were left over from our earlier internal experiments. We did not update those settings before importing the plots into the paper. We’ve now replaced all three panels and matched both axes to the 5% setup. The new horizontal axis counts optimizer updates from the OLMo-1B starting checkpoint. The vertical axis shows the score on each benchmark. Each update uses 1M tokens. All three panels end at the 5% results in Table 1.

PanelTraining tokensFinal updateSDV score in Table 1
(a) Math660M66033.7% GSM8K
(b) Code900M90031.2% HumanEval pass@1
(c) Biomedicine180M18032.4% MedQA

Figure 12. Table 1 uses 19.8% for uniform random selection at the 5% math budget. SDV reaches 33.7% at the same budget. The absolute difference is 13.9%. The red curve in Figure 12 subtracts the same 19.8% random score in every panel. Each setting uses the same starting checkpoint and 660M-token training budget.

Is there direct evidence that the saturation profile predicts diminishing learning gains beyond what can be explained by exposure counts or gradient magnitude? A small controlled analysis distinguishing consistent learning signals from conflicting or noisy signals would be informative.

R4. Removing saturation lowers GSM8K from 33.7% to 27.6% in Table 2. To test later learning directly we’ll compare low- and high-saturation candidates at the same checkpoint. We match exposure count, example length, gradient norm, and weighted energy per token. Both groups get the same training budget. We measure the loss drop on a separate validation set. We report the results for consistent, mixed, and cancelling gradients separately.

AuthorsOpenReview ↗

Response to Reviewer P4sM (3/6)

Thanks for asking us to match the generator and spell out the scoring cadence. Thanks also for highlighting state-dependent data value and the shared rule connecting selection with targeted synthesis.

Follow-up on Q1–Q2. Equal-budget generation and refresh accounting.

R5. Both synthesis arms train on the same real examples. We change the directions used to guide the generator. The event table tracks a different question. It shows when we refresh the basis and when we merely update scores on the current basis.

Table 1. Controlled generation at 660M consumed tokens, GSM8K.

PipelineGenerated (M)Accepted and used (M)Real consumed (M)Correct counts, three seedsAccuracy mean ± SD
SDV selection only00660422 / 428 / 43132.37% ± 0.35%
SDV selection + FAC-guided synthesis4020640430 / 434 / 43732.88% ± 0.27%
SDV selection + spectral synthesis4020640441 / 445 / 44833.71% ± 0.27%

Both generation runs use Qwen2.5-72B-Instruct with temperature 0.9 and top-p 0.95. They share seed topics and output cap and deduplication and acceptance. Only target guidance in the prompt changes. Pick the 640M real tokens once and give both arms those tokens. Each also gets 20M accepted generated tokens. Use one pass and 1M global tokens per update. Share three training seeds and keep per-item answers. The selection-only run uses the same real-data order and adds 20M real tokens.

Table 2. How often each method refreshes or updates scores over 660 training updates.

Curation armGradient-feature passesBasis refreshes after initWithin-basis repricing events
Random000
LESS3 saved checkpoints00
EK-FAC1 fixed checkpoint00
FAC feature selection0 (one activation pass)—65
SDV2 windows164

SDV refreshes the basis once in this setup. That happens after update 500. We select data once for every ten optimizer updates. This gives 66 selection blocks over 660 updates. The first block under each basis uses the initial scores for that basis. The other 64 blocks update scores on the same basis. FAC uses one set of initial scores and updates coverage for the other 65 blocks. A score computed after the last block is never used for selection.

Revision. State the default refresh cadence next to the loop and put generator, acceptance and consumed-token settings in the ablation caption.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear AC and Reviewers,

Thank you for reading SDV and taking the time to discuss the paper with us. P4sM highlighted the shared rule for data selection and synthesis and the broad evaluation. fF3Q noted the value of the dynamic score and the promising results. thNx highlighted the clear presentation and the tests of each component. We appreciate these comments and the specific suggestions for the revision.

Manuscript revisions. We’ve added the main explanations to the paper and marked the changes in blue. Appendix J gives the full protocols and tables. The table below shows how we used the feedback from the reviews and our replies.

#LocationWhat we addedComment
1§4.1 and Appendix J.3We define saturation as supplied squared-gradient energy. Two opposite projections add energy and have a signed sum of zero. We explain the fixed-basis utility beside this example.P4sM Q4 and fF3Q
2§5.2 and Appendix J.3The learning test matches past exposure and example length. It also matches gradient norm and weighted energy per token. We measure later loss reduction on a separate validation set.P4sM Q4 and fF3Q
3§5.2 and Appendix J.3We compare lower- and higher-saturation candidates under the same training budget. The tables give paired loss differences and intervals for consistent, mixed, and cancelling gradients.fF3Q follow-up and P4sM Q4
4§5.3 and Appendix J.4Selection alone scores 32.4% on GSM8K. The matched generation test starts each guidance arm with 640M real tokens and uses the same generator. Each arm adds 20M accepted tokens. We also give the random-guidance result and the share of tokens sent to high-price directions.P4sM Q2
5§5.1 and Appendix J.2SDV refreshes its basis every 500 optimizer updates and updates candidate scores after each selection batch. We give the scoring schedules for LESS, EK-FAC, and FAC. The appendix explains cached reprojection and compares it with an exact reference.P4sM Q1 and thNx
6§5.1 and Appendix J.1We name the model and corpus for each training regime. We define the candidate pools and learning windows. We give the domain splits and the training settings.thNx
7§5.3, Figures 4 and 12, and Appendix J.5Figure 4(a)–(c) now use the update counts and scores for the 5% setup. The curves end at 660 updates for math, 900 for code, and 180 for biomedicine. Their scores match Table 1. Figure 12 uses the same 19.8% random score in all four panels. The default SDV score is 33.7%. We also explain unique tokens and training consumption.P4sM Q1 and Q3
8§5.5 and Appendix J.6We give the full-pool cost calculation and the cost of each stage. These include gradient scans, basis refreshes, cached score updates, generation, and training. One full math scan takes about 242 node-hours at the reported scoring rate.fF3Q follow-up
9Appendix J.3 and J.6We compare exposure count, gradient norm, energy without saturation, and the full price against the same later loss outcomes. We also vary the scan budget at a common development target and give the total cost.fF3Q additional experiments

Our replies give the main settings and results for each question. The revision puts these explanations beside the method and experiments. Thank you again for your careful feedback and continued discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.