- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces a token-weighting objective for long-context mid-training. The high-level idea is that most tokens can be predicted from nearby context; so identifying those small amount of tokens which really benefit from long context can provide better training signal (also speedup training and consume less GPU memory). The authors propose a cheaper solution, comparing to previous work such as LongCE, consisting of three steps:
- segment input into non-overlapping segments and calcualte each token's SegRatio score by comparing the loss when full context is available against only local context is available; 2) filter out those easier tokens whose full-context loss is too small; 3) re-weight kept tokens not by their raw score but based on their rank percentile.
The experiments were conducted on three model families (dense and MoE models with up to 125B total parameters) and using training contexts up to 512K.
Results show than the proposed method is better than standard cross-entropy loss (treating all tokens equally) and LongCE.
Critical strengths
- Strengths
- the proposed method makes sense and is lighter (evidenced by faster training and lower GPU memory usage) than existing token weighting methods
- The authors tested three base architectures, convering both dense and mixture-of-experts models; however, the details of MOE are not provided
- The proposed method shows better performance than standard cross-entropy loss and other token re-weigthing technique (LongCE).
- the authors open-source their implementation
Critical weaknesses
- Weakness
- The full comparison (against Vanilla CE and LongCE) is only done on dense model; but not on MOE (authors mention LongCE implementation runs out of memory)
- Although the proposed method sounds interesting, it introduces a large number of hyperparameters, and the current ablation study is not very informative.
- I think the presentation could be further improved by using simpler language (rather than introducing new terms without explanation), summarizing key findings (instead of listing results), reducing the bold formatting, and making tables more compact.
Questions for authors
- Minor comments:
- Line 51: dpendencies (Wu, Deng, Chen).