- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the answer distribution of LLM during repetitive sampling at test time. The authors did a PCA analysis of the truncated probability distribution of answers and fitted a IRT type theory to study the ability-difficulty relation between LLMs and questions. They found a transition of probability spreading when questions are hard to a concentrated distribution when questions are relatively easier.
Critical strengths
-
The authors systematically studied the test-time sampling distribution, which is not well studied beyond aggregated metrics, such as entropy.
-
The authors tested their findings over models and datasets.
-
The figures are pretty.
Critical weaknesses
-
The writing is hard to follow. Many important details about the experiments are either missing or deferred to appendix. Many of them are essential to understanding and should be in the main text.
-
It is unclear to me what insight is here for the transition in Sec. 4.2. Does it just tell that when the problem is hard, the model will randomly guess and when the problem is getting easier, the model's probability of answering it correctly increases?
-
The sigmoid in the Rasch model introduced a transition as the inductive bias. It is unclear whether the transitions the authors found is new or is this just a loophole of fitting the difficulty score r with a curve already has the transition baked in.
-
The PCA analysis is very isolated from the other parts of the study and it is unclear what insight is there from the PCA part.
-
Some important related works are not properly discussed. For example, IRT has been used to study LLM ability in many different ways (IRT-router: Effective and interpretable multi-LLM routing via item response theory, ACL 2025; Lost in benchmarks? rethinking large language model benchmarking with item response theory, AAAI 2026; A Geometric Lens on LLM Abilities through Joint Embedding Item Response Theory, TMLR 2026).
Questions for authors
-
How is the PCA constructed for fig.1? Is this just stack all traces of all question-model pairs together and do the PCA? If so, is this a genuine effect or is the first few PCs simply driven by variance between models, or variance between benchmarks?
-
Can the authors show the shared geometry in Fig.1 is general instead of just an artifact of this set of models and benchmarks? It is unclear to me how to do it but I believe it is necessary if authors want to claim that a meaningful shared geometry emerges.
-
What is the ansatz of F(r) when fitting? Is that linear, MLP or is this a specific function form?
-
From the distribution of fig.2 and fig.3, together with the PCs in fig.1, it doesn't seem even p2 and p3 matters for this analysis. Is this simply a transition from all other answers to the most dominant answer? If so, what is new beyond just using the entropy/confidence?
Dear Reviewers,
Thank you for your time and effort in reviewing this paper. The authors have posted responses to your reviews. If you have not yet done so, please read their responses and leave a follow-up comment visible to the authors, indicating which concerns have been addressed and which remain unresolved.
Please also update your review and rating if your assessment has changed. Your feedback will help the authors address any remaining questions during the discussion period.
Thank you!