- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes an analysis of the next token prediction ability as models size grows. They show that larger models have a higher probability to generate the correct next token compare to smaller models, but the gain is not uniform across all instances: on a small but significant amount of instances, they have a higher probability than smaller models to generate a wrong next token.
Critical strengths
The paper proposes an interesting angle that goes further the standard story that "larger models are better", and rather investigate where they are better and where they are not.
Up to section 6 I find the paper easy to read.
Results are consistent and convincingly presented.
Critical weaknesses
- My first remark is on the chosen task. Overall, results are on the "next token prediction task". While an interesting diagnoses, I am unsure the conclusions can be extended to more operational settings.
- Also, the paper implies that a model predicting a "wrong next token" is "wrong", but without further analysis, we cannot infer the model's overall ability beyond this simple task (churns tokens can correspond to noisy data, position with several valid continuations etc).
- So I think the overall scope of the paper is much more narrow that what the narrative seems to imply.
- Section 6 current positioning is unclear. It comes after the main results, with a list of theoretical results, general discussions and idea suggestions with no experiments in the main paper. From what I understand, it seems to converge to Speculative Decoding idea without really naming it. In the current form, it is unclear what this section brings.
Overall, I think this paper main contribution is to count over several datasets the positions where models disagrees on the next token prediction task. Without further analysis and practical implication I am unsure this clears the bar for ICLR.
Questions for authors
See weaknesses.
9 comments
Thank you for your response, and we apologize for the length of our previous replies.
We tried to address your concerns in detail, but may have made our response difficult to follow.
To briefly summarize:
- Beyond next-token prediction: We evaluated seven full-answer benchmarks across 161 comparisons.
- Correctness: We conducted a blind audit of 2,800 continuations to distinguish token mismatches from actual errors.
- Main contribution: We showed that retraining explains much of the prediction churn, while scaling mainly changes its direction.
- Practical relevance: We clarified how our approach differs from speculative decoding and tested routing on 145 configurations with measured costs.
We would appreciate it if you could point out any specific concerns that remain, particularly regarding soundness, novelty, or practical relevance.
We are happy to conduct further experiments or analyses during the discussion period and look forward to your feedback.
Thank you again for your time and consideration.
Summary for our previous replies
We also wanted to briefly summarize the key results from our additional experiments.
- Task-level correctness: Across seven full-answer benchmarks and 161 comparisons, larger models break 1.0–11.4% of task instances, despite overall improvements in 157/161 comparisons. We also audited 2,800 continuations to examine valid alternative tokens.
- Beyond counting disagreements: Independent pretraining runs show that retraining accounts for 73–85% of token-level churn, while scaling primarily changes its direction. This distinction is the main finding of our analysis.
- Practical relevance: Unlike speculative decoding, our routing method can retain the smaller model's correct answer. It outperforms confidence-based routing in 52/58 math configurations. In 27/145 configurations, it matches or exceeds the larger model's accuracy with lower GPU cost.
These results and their experimental details are provided in W1–W4.
We hope these additional results help clarify the paper's contributions and address your concerns.
Very dense and confusing rebuttal
Dear authors, I thank you for your answer. However, I find the current answer extremely long and confusing (9 successive 5000 characters long comments). It features lots of new results, with lots of technical jargon and unclear interpretation.
As it stands, I cannot assess if my concerns are solved. I keep my original scores. Maybe if answering my concerns necessitates that much new textual content, a re submission might more relevant.