- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work proposes and tests a training-time auxiliary loss called OFF-DIAGONAL ATLAS TRANSPORT (OAT), designed to align projections of sampled pairs of past and future tokens, in addition to regular next-token prediction. OAT is compared to vanilla next-token prediction and a recent method Semantic Tube Prediction, and the work found improvements over NTP across 3 instruction-tuned model families in low-data fine-tuning settings, with advantages over STP on GSM datasets.
Critical strengths
- This writing is very clear and generally easy to follow.
- The experiments are comprehensive, covering multiple model families and datasets, ablations, cost tradeoff, and exploration of training-from-scratch beyond finetuning.
- This work opens some interesting questions, e.g. what was learned in the atlas.
Critical weaknesses
- While the methodology and quantitative results are described thoroughly, I find the paper a bit thin on the speculation and discussion on the behavioral intuition of why this method worked. That could be useful for a broad audience. E.g., if some behavioral analysis could offer specific insights on what types of temporal relationship/alignment are better preserved under OAT, or why OAT supported math reasoning better, that would make the work much more interesting.
Questions for authors
- Have you considered interpreting what was learned in the atlas/frames?
- How might OAT scale with larger models or longer sequences? Curious to hear your intuition on if the cost efficiency holds when the model scales up e.g. when more token pairs are needed to realize the benefit.
We have decided to withdraw our submission after further consideration. We sincerely thank the reviewers and Area Chair for their time and constructive feedback.