- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper focuses on test-time training (TTT) as an effective mechanism to improve long-context modeling performance in LLMs. It identifies the quality of context spans as a key factor in the effectiveness of TTT by showing significant performance gains of TTT on oracle spans vs random spans. Based on this observation, the authors propose self-guided TTT, which lets the LLM select important spans itself and conducts TTT on the selected ones. Experiments on long context benchmarks show improved performance when applying the proposed TTT method.
Critical strengths
- The motivating question of what data we should TTT on is a great one and the study in table 1 gives a clear reason for the study of the paper.
- The method is simple and builds on existing work of TTT, which can be easily incorporated into existing LLM workflows.
Critical weaknesses
The most important limitation of the current work is omission of key baseline methods. For example, a simple baseline is to simply append the spans selected by the proposed S-TTT method to the LLM’s context. This requires no training and is a simple inference-time method. Without this baseline, it is very hard to show the necessity of TTT, which is much more expensive than pure inference. Another baseline method could be to apply q-TTT on top of spans selected by the LLM, which could avoid the large cost of the proposed method.
Questions for authors
How well does the simple baseline described above perform compared to the proposed S-TTT method?