- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents a new method for filtering and selecting data during language model training, called Spectral Data Valuation. At a high level, SDV uses the geometry of the model's Fisher Information during training to approximate candidate data's value at a given model state.
The paper cleanly answers four research questions:
- Does SDV predict data value?
- Does SDV improve training efficiency?
- Do SDV prices generalize?
- What do SDV components contribute?
Overall, the paper found that the SDV method was effective in predicting how much a model would learn from candidate data and increased training efficiency compared to other data selection methods--in particular they showed that recomputing the SDV price during training was important in gaining this efficiency. They also showed that SDV prices generalized across model scale and data mixtures, and that the saturation component of the SDV computation had the most effect on performance--determining what data the model had already seen produced a meaningful change in the selected data and the training performance.
Critical strengths
- This paper is very well-written and well-organized. Even though I was not very familiar with data selection methods or the math used in computing the SDV metric, I had no trouble following the logic of their methods or their conclusions.
- The experimentation is cleanly laid out and thoroughly tests the value of their methods from several different angles. The decomposition of the SDV components was a particularly nice touch in demonstrating thoroughness.
- Figures and tables are well-labeled and clearly support the discussion in the main text.
- To the best of my knowledge, the methods and results here present a meaningful and novel contribution to the field.
Critical weaknesses
A few important experimental details were left out or only included in the appendices, see below.
Questions for authors
- I would have liked to see some discussion of the data and training methods in the main paper, as those were questions that kept coming up while I was reading. The coverage in the appendices was complete, but basic details would have been helpful in the main text
- I wasn't able to find details about how often or when the spectral probe was refreshed.
6 comments
Response to Reviewer thNx (2/3)
Thank you for asking for the basic setup in the main text. We appreciate your comment on the component tests across the three training regimes.
Follow-up on reproducible settings and the basis refresh.
R2. The 5% math curve in Figure 4(a) gives a concrete example. Each optimizer update consumes 1M tokens. The model has seen 500M tokens at the first basis refresh and 660M tokens at the end.
Table 1. How the basis changes over the 5% math run.
| Optimizer update | Training tokens used | What happens |
|---|---|---|
| 0 | 0M | Score candidates on the starting basis. |
| 500 | 500M | Refresh the basis. Project recent cached gradients onto the new directions and use the covariance summary for older gradients. |
| 660 | 660M | Finish the run. The last 160 updates use the new basis. |
The code and biomedical curves in Figure 4 use the same 1M-token batch. Code has 900 updates and refreshes once. Biomedicine has 180 updates and keeps its starting basis for the whole 5% run. SDV updates candidate scores after each selection batch in both runs.
We keep 64 directions per layer and use a factor EMA of 0.95. Continual pre-training uses the pilot stream for the starting factors and saturation. From-scratch training starts with zero saturation. All three training regimes use AdamW and a context length of 2,048. Each setting uses three seeds.
Revision. Section 5.1 gives the refresh rule and the math example. Figure 4 labels the update axis and the token budget for each panel. Appendix J.2 explains the update schedule in more detail.
Response to Reviewer thNx (3/3)
Thank you for taking a close look at how the probe is updated. We appreciate your kind words about the component analysis. We added a comparison on the refresh itself.
Additional experimental results. How accurate is cached reprojection after a basis refresh?
R3. At the first refresh we compare two ways to reuse old gradients. One uses only the recent cache. The other also uses a covariance summary of older gradients. We save all gradients in a short training interval and project them onto the new basis to get the reference. We use the same basis and the same 4,096 held-out candidates for all three profiles in each regime.
Table S1. Reprojection at update 500. The arrows go from the recent cache alone to the cache plus the summary.
| Training setup | Training tokens at update 500 | Price rank correlation | Mean absolute saturation error |
|---|---|---|---|
| From scratch | 250M | 0.91 → 0.96 | 0.071 → 0.036 |
| Continual math | 500M | 0.87 → 0.94 | 0.085 → 0.044 |
| Instruction tuning | 125M | 0.90 → 0.95 | 0.078 → 0.039 |
In the math comparison the order of the scores gets closer to the reference. The rank correlation goes from 0.87 to 0.94. The saturation error drops from 0.085 to 0.044. We see the same pattern in the other two setups.
Manuscript addition. Section 5.1 explains what we keep from past batches at a refresh. Appendix J.2 gives the reference comparison and the results for all three setups.
Response to Reviewer thNx (1/3)
Thank you for reading the paper so carefully. We appreciate your kind words about the way we explain SDV and test its parts.
R1. We refresh the spectral basis every 500 optimizer updates. A refresh gives us a new set of directions for measuring gradients. Between refreshes we keep the same directions. We update the scores of the remaining candidates after each selection batch. The 5% math run lasts 660 updates. It refreshes at update 500 and uses the new basis for the final 160 updates.
The candidate pool is the data we can score and choose for training. A learning window starts at the checkpoint for scoring. It ends at the next loss evaluation. We compare the scores at the start with the loss drop over that window.
Revision. Section 5.1 now gives the data and the training setup in the main text. It defines the learning window and gives the refresh rate in a short paragraph on scoring. Appendix J.1 lists the data splits. Appendix J.2 explains how we reuse gradients after a refresh.