- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SNAS for selecting long-context training documents. A calibration stage identifies attention heads that respond to useful distant information. These heads then score documents in one forward pass. Experiments report quality comparable to LongFilter with higher scoring throughput.
Critical strengths
- The method offers a useful quality–cost trade-off, reporting about 1.8× scoring throughput while maintaining comparable downstream performance.
- Matched training budgets, head-selection ablations, and calibration checks support the proposed scoring signal. Results cover multiple scoring models and data domains.
Critical weaknesses
- The method offers a useful quality–cost trade-off, reporting about 1.8× scoring throughput while maintaining comparable downstream performance.
- Matched training budgets, head-selection ablations, and calibration checks support the proposed scoring signal. Results cover multiple scoring models and data domains.
Questions for authors
- What is the total calibration-plus-scoring cost, and how large must a corpus be for SNAS to save compute?
- Can you report variation across runs for the main comparison and key ablations?
Response to Reviewer 5kDt (1/4)
Thank you for your positive assessment and the specific questions. We have uploaded a revised version of the paper, with all changes marked in blue. We group the comments into four parts and answer them in turn. Each part lists the corresponding weaknesses (W) and questions (Q) and states where the revised paper addresses them.
A. Calibration cost and the corpus size needed to recover it (W1, Q1)
Response. Thank you for raising this point. Reducing cost is the main contribution of the paper, so the calibration cost should be counted. We report the timing of the full calibration below.
Setup. Calibration and the scoring times in Table 2 use the same setup:
Calibration cost. The full calibration takes 49.0 GPU-h. It scans 1,002 documents and retains 987 (about 65M tokens). The repeated predictions at different context lengths run only on these ~1,000 documents and do not grow with the candidate pool:
The calibration data amount to less than 0.3% of the 30B-token candidate data in the paper.
Break-even scale. Calibration is done once per scoring model, whereas scoring is paid for every document. At the Table 2 throughput (SNAS 0.225 doc/s, LongFilter 0.125 doc/s), SNAS saves 3.56 GPU-s per document and recovers the calibration cost after about 50K documents (about 3.2B tokens at the average CC document length). Beyond that point it saves about 15 GPU-h per 1B tokens. The savings grow linearly with corpus size and reach about 15K GPU-h on a 1T-token corpus, which is a substantial cost advantage for data selection in large-scale long-context pretraining:
Reuse. The same head set HM is shared across CC, arXiv, and Books, and the calibration set itself mixes the three domains, so the three pools in the paper pay the calibration cost only once. Calibration can also be made cheaper: with 1/4 of the calibration documents (about 12 GPU-h), the resulting head set has a Jaccard similarity of 0.87 ± 0.06 with the full-calibration head set.
The revised paper reports the calibration cost in Section 4.4 (Scoring Efficiency), with the per-tier cost in Table 2(b). The timing setup, cost breakdown, and break-even analysis are in Appendix B.1.