acceptodds
Under review as a conference paper at ICLR 2027

Relative Capability Organizes the Geometry of Answer Distributions in Test-Time Scaling

Abstract

Test-time scaling naturally produces distributions over candidate answers, revealing how a model's responses vary across repeated samples on the same problem. While prior work has connected individual properties of these distributions to difficulty, uncertainty, and aggregation performance, whether the distribution as a whole exhibits a shared structure across models and tasks remains unclear. We find that ranked answer distributions exhibit a shared geometry beyond ranking and probability constraints, systematically organized by model-problem relative capability. As relative capability increases, distributions progress from fragmentation, through structured competition among leading answers, to single-cluster dominance. Correct-answer location is reorganized within this geometry: it shifts sharply toward the leading cluster in the intermediate regime, after which answer dominance increasingly coincides with correctness. This empirical law generalizes to held-out models and tasks and extends to constrained answer spaces through a systematic geometric deformation. Together, these findings offer a new structural perspective on test-time scaling, revealing systematic organization in LLM reasoning behavior across models and problems.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Test-Time Scaling, LLM Reasoning, Answer Distributions, Self-Consistency

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tVCwOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the answer distribution of LLM during repetitive sampling at test time. The authors did a PCA analysis of the truncated probability distribution of answers and fitted a IRT type theory to study the ability-difficulty relation between LLMs and questions. They found a transition of probability spreading when questions are hard to a concentrated distribution when questions are relatively easier.

Critical strengths
  • The authors systematically studied the test-time sampling distribution, which is not well studied beyond aggregated metrics, such as entropy.

  • The authors tested their findings over models and datasets.

  • The figures are pretty.

Critical weaknesses
  • The writing is hard to follow. Many important details about the experiments are either missing or deferred to appendix. Many of them are essential to understanding and should be in the main text.

  • It is unclear to me what insight is here for the transition in Sec. 4.2. Does it just tell that when the problem is hard, the model will randomly guess and when the problem is getting easier, the model's probability of answering it correctly increases?

  • The sigmoid in the Rasch model introduced a transition as the inductive bias. It is unclear whether the transitions the authors found is new or is this just a loophole of fitting the difficulty score r with a curve already has the transition baked in.

  • The PCA analysis is very isolated from the other parts of the study and it is unclear what insight is there from the PCA part.

  • Some important related works are not properly discussed. For example, IRT has been used to study LLM ability in many different ways (IRT-router: Effective and interpretable multi-LLM routing via item response theory, ACL 2025; Lost in benchmarks? rethinking large language model benchmarking with item response theory, AAAI 2026; A Geometric Lens on LLM Abilities through Joint Embedding Item Response Theory, TMLR 2026).

Questions for authors
  • How is the PCA constructed for fig.1? Is this just stack all traces of all question-model pairs together and do the PCA? If so, is this a genuine effect or is the first few PCs simply driven by variance between models, or variance between benchmarks?

  • Can the authors show the shared geometry in Fig.1 is general instead of just an artifact of this set of models and benchmarks? It is unclear to me how to do it but I believe it is necessary if authors want to claim that a meaningful shared geometry emerges.

  • What is the ansatz of F(r) when fitting? Is that linear, MLP or is this a specific function form?

  • From the distribution of fig.2 and fig.3, together with the PCs in fig.1, it doesn't seem even p2 and p3 matters for this analysis. Is this simply a transition from all other answers to the most dominant answer? If so, what is new beyond just using the entropy/confidence?

Reviewer_n1oGOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper mainly studies the test-time scaling from the lens of relative capability. They define a hardness variable and a competence score for the model, and they apply a transformation which computes something called relative capability, which is analogous to how much the model can easily answer the question.

Under that lens, they viewed the probability space generated by test-time scaling and projected it within a PCA dimensionality reduction. They show that this probability distribution over the possible answers starts from a uniform distribution, which they call fragmentation. Then a couple of more plausible answers raise their probabilities, and then the leading answer starts to get concentrated. This happens by moving from a hard problem to an easy problem.

The hard problem and easy problems are also the result of clustering various Rs and looking at the centroid of each cluster.

I think the main takeaways from this paper are:

  1. The ranked probability space of test-time scaling has a structure beyond only random ranked probabilities.
  2. The relative capability itself can identify various degrees of hardness of a problem for a model.
  3. How the probability of the top answer/the true answer shifts by moving from the hardest problem to the easiest problem in terms of relative capability. They show at what degrees of hardness (again, in terms of R) self-consistency can be a good measure of quantifying the correctness of the output (this is in case I understood the paper correctly on that part).
  4. They showed how multi-choice, instead of open-ended, question answering could project the probability space within the PCA reduction.
Critical strengths

This is an empirical study. I don't expect too much theory or comparison with any baselines. As an empirical study, I find this paper systematically discovering meaningful relationships within the data and not just reporting some summary statistic. I think definition of hardness (relative capability), the comparison with baseline ranked probabilities, and several other experiments (like projecting from an open-ended answer to constrained answer space) were quite novel, and they helped to find meaningful relationships within the data.

The empirical findings from this paper could later be the root of many other studies on building a better inference policy, or a better uncertainty quantification, or a better harness.

Critical weaknesses
  • W1. The "law" (stages of relative capability) is shown only through bin means; the variance of the clusters is never shown. Figure 2B gives mean p1..p5, rest per r-bin and nothing about variance. The held-out L1 of 0.453 (lines 226 to 228) looks large if most mass sits in p1, p2 and rest; whether the law is informative depends on how scattered distributions are within a bin, which is not reported. I would appreciate if the scatter inside each cluster is also reported. Without that I can not infer any general distributional judgement.

  • W2. The contrast between Fragmentation versus structured competition is not yet clear. The three regimes are named from the mean curves, but nothing quantifies where fragmentation ends (e.g. TV or KL distance to a uniform categorical, or to the ranked synthetic null). Maybe this comparison could help to better distinguish these phases.

  • W3. "Transferability" in Section 5 is not a test-time quantity. r is estimated from correctness (cross-fitted Rasch), so predicting the shape from r requires prior evaluation of the model on the problems (and also for some other models in order to compute the difficulty of the problem). It is unclear what the Section 5 tables establish beyond the held-out model or task sharing the same hypothesis family; the practical meaning should be stated.

  • W4. The constrained-answer analysis (Sec. 6, lines 396 to 400): besides the finding of the paper, one can argue that giving the choices removes most wrong answers from the space, so multiple choice is simply easier, and the deformation may reflect the model having seen the candidates rather than a geometric projection. The direction of the comparison (open to constrained, instead of also treating GPQA as open) is incomplete: I also suggest the authors explore a hard multiple-choice problem presented to the model as an open-answer problem, and see if the projection vanishes and the corresponding vectors scatter in the entire space again.

Questions for authors

Q1. Please consider adding a companion to Figure 2B that shows variance per bin, and a figure that picks three or four random cases from one bin at different positions and plots their top-3 or top-4 probabilities on a simplex.

Q2. Is the rank-6-to-rank-1 pattern specific to LLM outputs, or would any generation policy with a given accuracy produce it? Can it be derived analytically for a generic stochastic answerer, and how does the matched null in Sec. 3 relate to that? My question at a higher level is whether this is in the nature of question answering or whether it is only considering the LLM architecture. I do not expect the authors to do experiments for it. I jsut ask what additional insight do they have for this question?

Q3. How can we contrast fragmentation from the structural competition? Can TV or KL divergence of the empirical distributions and to the synthetic distributions help with for that?

Q4. (lines 311 to 315). Do the authors claim that self-consistency fails to quantify uncertainty specifically at intermediate r, i.e. when the model is neither clearly capable nor clearly incapable? If so, please state it explicitly, and discuss it; it is one of the more interesting results.

Q5. What does transferability mean operationally here? Is it only that a held-out model or benchmark lies in the same hypothesis family? The question basically takes this fact into account: the relative capability is computed over problems and models in a cross-computed manner.

Q6. I cannot fully distinguish whether this projection of constrained-answer questions is because of limited dimension of the probability space or if a constrained answer is generally easier than an open-ended answer (for any LLM). This could potentially be resolved if the authors could do the reverse experiment as well, which means removing the multiple choices from the questions and seeing again how the probabilities would form in the space.

Reviewer_3KeCOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper investigates the structure of answer distributions generated by repeated LLM reasoning samples during test-time scaling. The authors show that ranked answer distributions exhibit a shared geometric structure that is systematically organized by model–problem relative capability, estimated using a Rasch model. As relative capability increases, answer distributions transition from fragmented responses to competition among leading answers and eventually to single-answer dominance. The paper further analyzes how correct answers become increasingly concentrated in the dominant cluster, explaining certain failures of self-consistency. Experiments across seven LLMs and seven reasoning benchmarks support these observations, with additional evaluations on held-out models, benchmarks, and constrained answer spaces.

Critical strengths

The paper provides an interesting empirical framework connecting model–problem relative capability with the overall structure of answer distributions, going beyond individual statistics such as entropy and majority agreement.

The paper proposes a modeling method rather than a method for improving LLM inference.

Critical weaknesses
  1. Limited practical contribution: Although the paper identifies a relationship between relative capability and answer-distribution geometry, it does not demonstrate how this relationship can improve test-time scaling, reasoning accuracy, or sampling efficiency. The practical value beyond descriptive analysis remains unclear.

  2. Dependence on correctness-based capability estimation: The proposed framework relies on relative capability estimated from correctness observations across models and problems. It is unclear how the framework can be applied to unseen problems without access to such information. Moreover, the paper does not sufficiently establish whether the observed geometric relationship provides additional explanatory value beyond existing difficulty or answer-concentration measures.

Questions for authors
  1. Practical applicability: How can the proposed geometric relationship be used to improve test-time scaling or answer selection? Can the authors demonstrate a concrete improvement over conventional self-consistency?

  2. Estimating relative capability: How can relative capability be estimated for a new problem without using correctness observations? Does the proposed relationship remain predictive when capability is estimated from answer-distribution statistics alone?

  3. Novelty of the geometric relationship: Since answer concentration and diversity are already known to correlate with problem difficulty, what additional information does the proposed geometric representation capture beyond simpler statistics such as entropy, top-1 probability, or the top-2 probability gap? Can the authors compare predictive performance against baselines using these statistics?

  4. Generalization: The proposed law fails to improve prediction over the baseline for two of six model conditions on HMMT25. What characteristics of these cases explain the failure, and what does this imply about the claimed generality of the relationship?

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your time and effort in reviewing this paper. The authors have posted responses to your reviews. If you have not yet done so, please read their responses and leave a follow-up comment visible to the authors, indicating which concerns have been addressed and which remain unresolved.

Please also update your review and rating if your assessment has changed. Your feedback will help the authors address any remaining questions during the discussion period.

Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.