acceptodds
Under review as a conference paper at ICLR 2027

BEYOND GROUP SIZE : THE SPECTRAL GEOMETRY OF PLACKETT–LUCE SUPERVISION

Abstract

LLM post-training increasingly uses ranked groups of candidate responses rather than isolated pairwise choices, raising the question of how much local Fisher information one complete-ranking observation carries about the underlying relative scores. A complete ranking is not equivalent to observing every pair independently: the Plackett–Luce model generates it sequentially, so a pair contributes only while both members remain unselected. We show that the exact Fisher operator of a complete ranking is a weighted graph on the candidates whose edge weights factor into two logically distinct terms. The first is classical Bradley–Terry discriminability, fixed by the pair's own score gap relative to the evaluator's resolution. The second we call exposure: it records how the sequential process attenuates that information, and it changes when the other candidates move even though the pair's own scores do not. We prove exposure never exceeds one and never falls below one part in the group size minus one. The exact operator is therefore sandwiched, in the positive semidefinite sense, between two multiples of the Laplacian of a reward-gap graph whose edges are heavy between candidates that are hard to tell apart, so the weakest identifiable Fisher curvature of a full ranking, the relative-score contrast it constrains least, is pinned by that graph's algebraic connectivity to within a factor of the group size minus one. Symmetry calibrates this picture rather than explaining it: there the curvature has a closed form, rising with group size but saturating while ranking entropy and total Fisher mass keep growing, and stagewise curvatures provably cannot be added. Group size and reward range therefore do not determine this curvature, and we test total Fisher mass empirically. Across 9,696 exact instances the bounds hold without genuine violation; at fixed group size and range the curvature moves by a factor of 8384 while the trace moves the other way, and nearly trace-matched configurations differ hundreds-fold. These are local information-geometric results, not finite-sample claims.

Primary area:
learning theory
Keywords:
preference learning, LLM post-training, Plackett–Luce model, listwise ranking, Fisher information, information geometry, spectral graph theory, identifiability, ranking models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pNSKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper considers a scenario in which a group of ranked items is given, and we seek to learn the underlying (true) scores that describe those preferences. For this, the paper studies the local Fisher information, which a single complete Plackett-Luce ranking of the K items provides about the true underlying scores. They base their analysis on the smallest (non-zero) eigenvalue of the Fisher information γPL\gamma_{PL}. They show that the Fisher matrix of the Plackett-Luce model is equivalent to a Laplacian, with edge weights being the Bradley-Terry discriminability times an exposure coefficient. Further, they obtain (Th. 5.5) a two-sided bound on γPL\gamma_{PL}, based on the reward-gap Laplacian algebraic connectivity of the Bradley Terry Fisher Information. Two main conclusions can be drawn: (A) For symmetric scores, increasing the group size K can only improve γPL\gamma_{PL} boundedly. (B) For non-symmetric scores, γPL\gamma_{PL} is restricted by the algebraic connectivity of the reward-gap graph between the underlying items. As such, scores that split into well-separated clusters are less beneficial than those that exhibit smaller between-cluster gaps.

Critical strengths

[S1] Theoretical Soundness: The theoretical derivations ( I have checked some but not all) seem sound and provide a detailed characterization of the problem. Furthermore, I find the factorization the strongest theoretical insight of the work.

[S2] Separability between trace: Central to this work, the authors convincingly show that the trace can convey insights different from those of γPL\gamma_{PL} (see Sec. 6.1). As such, it gives a basis why γPL\gamma_{PL} might be of practical benefit, yet this requires more in-depth and realistic evaluations (See [W1]).

Critical weaknesses

[W1] Lack of Practical Implication Investigation: With Corollary 5.7, a statement is provided on how to select different candidate groups of size K for practical applications, since computing the Bradley-Terry information is much cheaper than computing the complete Plackett-Luce Hessian. I find that such a statement should prompt at least small-scale experiments to demonstrate whether practical benefits can be gained from these theoretical insights. This would substantially benefit the paper.

[W2] Readability and Presentation: The paper is structured in largely self-contained sections, without a clear storyline connecting them. For example, I find the connection from symmetric to the asymmetric analysis hard to follow intuitively and would benefit from a more overarching storyline. The appendix (in particular Appendix H) seems to contain results from a previous submission. Table H.3 summarizes the results in Table 6 only qualitatively, whilst the latter is not referenced. In general, Appendix H is written to contain results as further evidence, yet H.4 and its Table 7 contain results noted as „secondary“ by the authors. The paragraph on „Prior validation code“, which probably suggests changes in the original code basis, is uninformative in the Appendix of a paper.

[W3] Related Work Comparison: As the authors have written, Hajek et al. already compared with the graph Laplacian and used the Cramer-Rao bound, which is closely related to the Fisher information. To my current understanding, this is conceptually close to the present work; therefore, I would appreciate a more detailed comparison of the differences between them. See [Q4].

Questions for authors

[Q1] How tight is the Spectral sandwich in Theorem 5.5? In particular, is the lower bound of 1K−1\frac{1}{K-1} tight, or could a tighter one be achieved?

[Q2] What would be the practical implications or recommendations based on Corollary 5.7?

[Q3] Given that the theoretical analysis is only restricted to a fixed complete ranking, how would the results change beyond that? Could you give insights here?

[Q4] Could you clarify to what extent your results differ from those of Hajek et al., as I currently understand that both papers deal with analyzing the Plackett Luce model via Fisher bounds based on the Laplacian.

3 comments
AuthorsOpenReview ↗

Clarification of Novelty and New Spectral Group-Selection Results

We thank the reviewer for recognizing the exact factorization and the trace-versus-weakest-curvature distinction. The revision responds to each weakness and question explicitly.

Critical Weakness 1 — No demonstrated practical benefit. We agree that downstream gains from spectral group selection remain untested. Cor. 5.7 is a proved approximation for the PL Fisher-curvature objective, not a learning-performance claim. New Appendix E provides an exact K=4K=4 two-group illustration: at R/σ=3R/\sigma=3, the spectral criterion chooses the group with 11.4 times the exact weakest-direction curvature of the trace-preferred group, even though the groups have equal size and range. The respective traces are 0.638/σ20.638/\sigma^2 and 1.010/σ21.010/\sigma^2. This demonstrates a decision consequence, not improved finite-sample estimation or LLM post-training.

Critical Weakness 2 — Storyline and Appendix H. The revised progression is: previously known symmetric baseline (Thm. 3.1) →\to proper-subset obstruction (Thm. 4.1) →\to arbitrary-score exposure factorization (Thm. 5.3) →\to spectral consequences (Thm. 5.5). We numbered equations (1)–(10) and added main-text proof guidance. Appendix H now states that the historical Pythia and regex-lite experiments used a trace-based, not spectral, selector; it includes unfavorable validation results and confidence intervals. Appendix I explains the prior validation-code distinction and its limits. Those historical results do not test Cor. 5.7, and the Discussion no longer presents them as validation of spectral selection.

Critical Weakness 3 — Comparison with Hajek, Oh & Xu (2014). We agree that the overlap is important. Their analysis of PL rankings on assigned subsets already gives pairwise stagewise Fisher terms and the equal-score calculation underlying our Thm. 3.1 (proof of their Thm. 2). The revised §3 explicitly credits this, and §7 and App. J sharpen the comparison. Their curvature comparison uses a score-independent co-occurrence Laplacian with constants depending on score range. Our contribution is not the generic Laplacian form: at arbitrary scores, Thm. 5.3 (6) separates each edge into BT discriminability and listwise exposure; Thm. 5.4 bounds exposure using the first-hit identity (7); and Thm. 5.5 (8) gives a score-dependent, range-free operator sandwich. Cor. 5.6 and 5.7 derive fixed-range separation and group selection from it. We do not claim novelty for the symmetric formula.

Question 1 — How tight is the sandwich? The universal 1/(K−1)1/(K-1) coefficient need not be sharp. For K=3K=3, eq. (9) gives αab=1−q+q2≥3/4\alpha_{ab}=1-q+q^2\ge3/4, hence 34LBT⪯Hσ⪯LBT\tfrac34L_{\rm BT}\preceq H_\sigma\preceq L_{\rm BT}. This 3/43/4 is sharp: with r=(0,0,−t)r=(0,0,-t) and contrast v∝(−1/2,−1/2,1)v\propto(-1/2,-1/2,1), the Rayleigh ratio approaches 3/43/4 as t→∞t\to\infty. The optimal general-KK constant remains open; numerical slack alone does not settle it.

Question 2 — What should a practitioner do with Cor. 5.7? For a finite family of equal-size groups, obtain target or estimated scores, build LBTL_{\rm BT} from pairwise gaps using (3), and select the group maximizing λ2(LBT)\lambda_2(L_{\rm BT}) before collecting rankings. Cor. 5.7 guarantees at least 1/(K−1)1/(K-1) of the best group's true PL curvature; for K=3K=3, 3/43/4 applies. Appendix L.2 (Prop. L.3) shows the factor becomes e−2δ/σ/(K−1)e^{-2\delta/\sigma}/(K-1) when every within-group estimated gap error is bounded by δ\delta. No practical reward-model accuracy is assumed without this condition.

Question 3 — Does the theory extend beyond one ranking? Yes, for fixed independent complete PL ranking observations on repeated or overlapping groups in a common score space. Appendix L.3 (Prop. L.4) proves additive Fisher information and

∑tLtKt−1⪯Htotal⪯∑tLt.\sum_t\frac{L_t}{K_t-1}\preceq H_{\rm total}\preceq\sum_t L_t.

The aggregate operators have the same kernel; identifiability depends on connectivity of the union graph. Adaptive selection requires a predictable history-based rule and is not an adaptive finite-sample guarantee.

Question 4 — Precisely how does this differ from Hajek et al.? The distinction is not whether PL Fisher is a Laplacian, nor the equal-score calculation: both precede us. It is the exact arbitrary-score edge factorization and comparison with a score-dependent BT reward-gap Laplacian, whose multiplicative constants are independent of reward range. That makes the listwise effect of third candidates explicit and supports Cor. 5.6–5.7. See revised §7 and Appendix J.

We hope the revised attribution and explicit limits clarify the novelty and significance of the proved results.

Thank you for your response, and I apologize for the late reply. I lost track of this.

I have read the rebuttal and the revised paper, and they have not convinced me. I therefore keep my original score and list the details of this decision below:

Empirical experiments [RW1]

There are still no experiments that show practical use. The new Appendix E is an exact calculation on two hand-picked groups. I would have wanted some real experiments demonstrating that the theoretical insights are meaningful to downstream users, or at least a rationale for why this is not necessary.

New storyline [RW2]

The changes are not marked (e.g., in blue), which makes it hard to follow what was revised. The paper has changed substantially, and I believe a version with this many changes needs a new round of review. I also don’t find it much easier to understand: the sections still read as largely separate pieces, and Appendix H still contains results from an earlier submission that you call secondary and that do not test the proposed criterion.

Benefit of the new insights [RW3]

It is still unclear what exactly the comparison with Hajek et al. is. The authors state the difference but do not show, formula by formula, what is new and what is a reformulation. In particular, how does Thm. 5.3 differ from Hajek et al.'s results? Appendix J adds no further insight here; you state yourselves that it only expands on §7.

AuthorsOpenReview ↗

Clarification of Theoretical Contributions, Revisions, and Prior-Work Comparison

We thank the reviewer for the follow-up and for taking the time to examine the revised manuscript. We address the three remaining concerns directly.

RW1 — Practical experiments.

The paper's contributions are theoretical: the exact arbitrary-score factorization of the Plackett–Luce Fisher operator (Theorem 5.3), universal exposure bounds (Theorem 5.4), a two-sided spectral comparison (Theorem 5.5), and a group-selection criterion with a provable approximation guarantee (Corollary 5.7).

These results characterize the information geometry of complete ranking observations. They do not establish that a particular LLM training algorithm benefits from the proposed selection criterion. Downstream evaluation remains an open empirical question, and we do not claim otherwise.

RW2 — Changes in the revision.

The research question and central results (Theorems 3.1, 4.1, and 5.3–5.5) are unchanged from the original submission. The revision adds equation numbering, main-text proof guidance, clearer attribution of prior results, and supporting appendices.

Appendix H was titled "Historical Supporting Experiments" in the original submission. It is retained for transparency and does not validate the proposed spectral selection criterion. We acknowledge that the revisions were not visually highlighted, making the changes harder to identify.

RW3 — Mathematical comparison with Hajek, Oh & Xu (2014).

The correspondence between the results is as follows:

ResultHajek et al. (2014)This paper
Stagewise Fisher structureProof of Theorem 2 (§3.2, arXiv version)Established result; credited in §3
Equal-score Fisher curvatureProof of Theorem 2 (§3.2, arXiv version)Theorem 3.1; established result
BT discriminability × listwise exposureNot isolated as a separate factorTheorem 5.3, eq. (6)
Universal exposure bounds, 1/(K−1)≤αab≤11/(K-1)\leq\alpha_{ab}\leq1No corresponding bounds identifiedTheorem 5.4, eq. (7)
Curvature comparisonCo-occurrence Laplacian (Definition 3); range-dependent eigenvalue bound (Lemma 2)Score-dependent BT Laplacian; two-sided operator bound (Theorem 5.5, eq. (8))
Fixed-range curvature separation and group-selection guaranteeNo corresponding guarantees identifiedCorollaries 5.6–5.7

The specific distinction in Theorem 5.3 is that the exact aggregate PL Fisher operator is expressed as

Hσ(r;r)=∑a<bαab(r,σ) κlog⁡(ra−rb;σ) Bab,H_\sigma(r;r)=\sum_{a<b}\alpha_{ab}(r,\sigma)\,\kappa_{\log}(r_a-r_b;\sigma)\,B_{ab},

where κlog⁡\kappa_{\log} depends only on the pairwise score gap, while αab\alpha_{ab} captures exposure within the full candidate list.

Theorem 5.4 establishes universal bounds on this exposure coefficient. These yield the two-sided operator comparison in Theorem 5.5:

1K−1LBT(r,σ)⪯Hσ(r;r)⪯LBT(r,σ).\frac{1}{K-1}L_{\rm BT}(r,\sigma)\preceq H_\sigma(r;r)\preceq L_{\rm BT}(r,\sigma).

The multiplicative constants 1/(K−1)1/(K-1) and 11 are independent of score range. The comparison operator retains score dependence through the pairwise discriminability kernel κlog⁡\kappa_{\log}.

Reviewer_NGqYOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper provides theoretical results on Fisher information matrix and its eigenvalues in the Placket-Luce ranking framework. The main result from the paper shows that the Fisher operator has an exact representation which is represented as a product of two terms indicating pairwise discriminability (which is a previously used quantity) and a list-wise exposure. A few other results are also shown. In general, the main takeaways from the paper are that when the scores of candidates are close to one another, they reveal more rather than candidates that are far away in terms of scores. For a pair whose scores remain fixed, its contribution gets determined by other candidates.

Critical strengths

A good attempt at an interesting theoretical work.

Critical weaknesses
  1. This paper is very hard to read and understand. For such a dense, mathematical paper, NO EQUATION is ever numbered either in the main paper or the Appendix nor is there any reference to any of the earlier equations, this is a first sign of an extremely poor writing.

2.Extremely poor and overloaded notations without adequate explanations throughout the paper (see the other comments below). The paper starts in an extremely dense way even in the introduction. Although, there is a setup and notation later on, it hardly provides much useful information. I think it would be better to reorganize the paper to keep the intro at a high level, introduce Setup and notation thoroughly with consistent and clear notations and then discuss the main results in the setup, framework and notation introduced then provide the details in later sections.

  1. Practical uses or applications of the theory is not elaborated on adequately. What would this paper really change? The authors' own admission in the conclusion seems to suggest this paper has no consequence on LLMs although it is motivated from LLMs in the beginning. With Corollary 5.7, the authors do state there maybe some practical consequence but shy short of going into the details.

  2. The paper is motivated with reference to LLMs with grouped rollouts which work with grouped rewards but not necessarily Placket-Luce rankings, the connection is at best thin.

  3. The full paper with appendix is about 23 pages with every single proof written in the Appendix with not even a proof sketch provided in the main paper.

  4. The experiments are simply numerical examples to confirm the theory on toy like problems rather than based on any real world examples.

Questions for authors

The introduction, rather than being at a high level and giving more of an intuitive explanation of the main results and their significance, readily goes into mathematical details without enough context.

On lines 50-52, it reads "Let r = (r1, . . . , rK) be the candidate scores; they determine a distribution over complete rankings." What is the setup here? How does a fixed ranking scores over K candidates "determine a distribution"? What are the assumptions made?

Lines 40-74: While the introduction gives a few generic citation papers, nothing is cited for a lot of things that are stated very vehemently on Fisher information, local sensitivity, how complete rankings and Fisher information are related etc.

Lines 59-60: What is the evaluator score \sgima?

S is very heavily overloaded in the paper with notations such as S_K, S_i(\omega), S_n

H is used with many subscripts with many of them not being properly introduced.

What is S on line 153? This S is not defined before.

While H_\sigma(r,r) is defined in line 142, It is not clear what H_\sigma(u,r) on line 157 is.

What is \kappa on line 161?

On line 173, what is H_{sym, \sigma}?

On line 176, what is \gamma_sym(K, \sigma)?

On line 172/173, given the definition of H_K why is not simply K/m since it is K times addition over 1/m?

I can keep listing such fundamental notational issues with this paper, but I think it is futile. The authors should thorughly go through the manuscript and focus on notational consistency and readability.

The bounds in the sandwich theorem is weak for the cases where K is large, but the experiments never go beyond K=6.

1 comment
AuthorsOpenReview ↗

Revised Mathematical Presentation and Clarification of Results

We appreciate the review. References below are to the revised PDF; original line numbers have shifted.

Critical Weakness 1 — No numbered equations. We numbered ten central displays (1)–(10), from the PL law and Fisher identity to the factorization, sandwich, and group-selection guarantee, and added cross-references.

Critical Weakness 2 — Dense introduction and overloaded notation. We clarified the PL observation model, fixed evaluator resolution, the symmetric-to-arbitrary-score transition, and the definitions in §2. The SS variants are distinguished below. We retained the paper’s structure.

Critical Weakness 3 — Unclear practical use. Cor. 5.7 (eq. (10)) is an actionable PL-information guarantee: among feasible equal-size groups, calculate the score-gap Laplacian (3) and choose the largest λ2\lambda_2. This guarantees at least 1/(K−1)1/(K-1) of optimal PL curvature. Appendix L.2 proves a conditional estimated-score extension. Appendix E gives an exact 11.4×11.4\times curvature example. No downstream learning improvement is claimed.

Critical Weakness 4 — GRPO is not PL ranking feedback. Agreed. Revised §1 explicitly distinguishes complete PL rankings from GRPO's group-normalized scalar rewards; our theorems apply to the former, not automatically to the latter.

Critical Weakness 5 — Proofs only in appendices. Section 5 now gives proof guidance: stagewise covariance averaging yields factorization (6); the first-hit identity (7) holds in expectation, not pathwise; Cauchy–Schwarz and Jensen give exposure bounds; PSD edge summation gives sandwich (8). The proper-subset nullspace obstruction (Thm. 4.1) explains why operators must be aggregated before taking minimum eigenvalues. Full proofs remain in Appendices B–D.

Critical Weakness 6 — Synthetic numerical evidence. The 9,696 exact instances check analytic identities via independent implementations, not LLM performance. Historical Appendix H used a trace-based selector, not Cor. 5.7's spectral rule. Downstream evaluation remains open.

Question — Introductory context and citations (original lines 40–74). The introduction now distinguishes ranking feedback from scalar rewards and explains the weak-direction Fisher objective. The population KL/Fisher identity is eq. (2); classical PL sources are cited in §1–2. Hajek, Oh & Xu (2014) are now explicitly credited for the equal-score Fisher calculation before Thm. 3.1 and compared in §7.

Question — How do fixed scores determine a ranking distribution (lines 50–52)? Scores are parameters, not a deterministic observed ranking. Conditional on rr and fixed σ>0\sigma>0, a random permutation is drawn sequentially with probabilities proportional to era/σe^{r_a/\sigma} among remaining items (PL law, eq. (1)). Only score differences are identifiable.

Question — What is σ\sigma (lines 59–60)? It is the fixed, known evaluator-resolution/temperature parameter. Probabilities depend on score gaps divided by σ\sigma (§2).

Question — What do the various SS symbols mean? SKS_K is the set of permutations; Si(ω)S_i(\omega) is the remaining candidate set at stage ii; SnS_n is a remaining set of size nn; and generic SS in a stage covariance is an active subset, now stated explicitly in §2. We acknowledge the overloading.

Question — What is Hσ(u;r)H_\sigma(u;r) (line 157)? It is ∇u2Lσ(u;r)\nabla_u^2\mathcal L_\sigma(u;r), the Hessian of population cross-entropy at model scores uu under true scores rr; §2 now defines it locally. At u=ru=r it equals Fisher information (2).

Question — What is κ\kappa (line 161)? κlog⁡(Δ;σ)=sech⁡2(Δ/(2σ))/(4σ2)\kappa_{\log}(\Delta;\sigma)=\operatorname{sech}^2(\Delta/(2\sigma))/(4\sigma^2), the direct BT comparison Fisher kernel (3).

Questions — Hsym,σH_{\rm sym,\sigma} and γsym\gamma_{\rm sym} (lines 173–176)? They are the equal-score Fisher matrix and its smallest identifiable eigenvalue, respectively (Thm. 3.1, eq. (4)).

Question — Why K−HKK-H_K, not K/mK/m? HK=∑m=1K1/mH_K=\sum_{m=1}^K1/m is the harmonic number; mm varies in the sum. Stage contributions yield ∑n=2K(n−1)/n=K−HK\sum_{n=2}^K(n-1)/n=K-H_K (§3; App. B).

Question — Are the bounds useful for large KK? The sandwich (8) holds for every finite KK but is not necessarily sharp. At symmetry the exact γsym=(K−HK)/(σ2(K−1))→1/σ2\gamma_{\rm sym}=(K-H_K)/(\sigma^2(K-1))\to1/\sigma^2 proves saturation; the sandwich alone does not. Its upper bound loosens linearly, while its lower bound is within a factor below 4 of the exact symmetric value. Enumeration stops at K=6K=6 due to factorial cost, not a theorem restriction. For K=3K=3 the sharp lower constant is 3/43/4; general-KK sharpness remains open.

We have not identified an invalid step in the central proofs. If a specific proof step remains in doubt, we welcome its identification.

Reviewer_rGjbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the Fisher information inherent in the Plackett–Luce ranking model, focusing on how the geometric structure of the candidates' scores influences the directions in the relative score space that are most difficult to discern. A key contribution is the exact decomposition of the PL Fisher information into the product of pairwise discriminability and list-level exposure coefficients. Furthermore, the authors propose a spectral sandwich theorem, which establishes a connection between the smallest non-zero eigenvalue of the Fisher matrix and the algebraic connectivity of a Bradley–Terry graph weighted by reward gaps. These findings are validated through precise calculations across 9,696 synthetic configurations, controlled experiments involving score perturbations, and comparative analyses based on trace matching.

Critical strengths
  1. This paper derives an exact decomposition of the Fisher information under the Plackett–Luce model, breaking it down into pairwise discriminability and a listwise exposure coefficient.
  2. This paper establishs a spectral sandwich structure, which reveals a connection between the Fisher curvature in the weakest identifiable direction and the algebraic connectivity of a Bradley–Terry graph weighted by reward gaps.
Critical weaknesses

1.The primary validation in this paper relies on synthetic scoring configurations with small candidate sets; it fails to demonstrate that the proposed spectral criterion improves preference learning or post-training outcomes for language models in realistic scenarios. In the Pythia-410M experiments, only three random seeds were used, and some confidence intervals—calculated via the paired-bootstrap method—include zero. The authors themselves acknowledge that the practical benefits of this approach for downstream tasks remain unverified.

2.The evidence linking local Fisher information to estimation efficiency in finite-sample settings is insufficient. In Appendix K, the initial hypothesis is not supported across all 60 experimental configurations: regression analysis yields a slope of −3.750 with an R² of 0.386. The expected relationship emerges only after excluding samples deemed to lack sufficient effective information (resulting in a slope of −1.0045 and an R² of 0.9955). The authors concede that this filtering criterion was introduced post hoc. While it is logically plausible that an estimator might fail under conditions of severe undersampling, the paper provides no independent justification for the chosen threshold, nor does it offer error guarantees for finite-sample regimes. Consequently, these experiments provide only limited support for any general conclusions regarding sample efficiency.

Questions for authors

1.Finite-sample estimation guarantees. Appendix K reports that the hypothesized relationship between estimation error and local Fisher information is not supported across all 60 experimental configurations, and the expected scaling emerges only after a post hoc restriction to adequately sampled cases. What independent theoretical justification supports the chosen threshold, and how sensitive are the results to alternative thresholds?

2.Robustness of the spectral group-selection guarantee. The proposed selection criterion depends on underlying reward scores, which are generally unknown in practical preference-learning settings. How sensitive is the guarantee to errors in estimated reward scores, and can the authors establish whether the approximation remains meaningful when these scores come from an imperfect reward model?

1 comment
AuthorsOpenReview ↗

Clarification of Theoretical Contributions and New Robustness Analyses

We thank the reviewer for recognizing the discriminability–exposure factorization and spectral sandwich. The revised manuscript separates proved information-geometric claims from empirical consequences and adds Appendix L to answer both questions.

Critical Weakness 1 — Limited practical validation. The primary contribution is analytical, not a new LLM training algorithm: Thm. 5.3 gives the exact arbitrary-score factorization (eq. (6)); Thm. 5.4 bounds exposure; Thm. 5.5 proves the range-free spectral sandwich (eq. (8)); Cor. 5.6 proves fixed-range curvature separation; and Cor. 5.7 proves a group-selection approximation (eq. (10)). These results follow from proofs, not synthetic regression. The 9,696 exact configurations independently check the identities and illustrate their consequences; they do not demonstrate downstream learning gains.

The Pythia-410M results have only three seeds and confidence intervals crossing zero. Appendix H now explicitly states that those experiments and the historical regex-lite runs used a trace-based selector, not the spectral rule of Cor. 5.7. They cannot establish spectral-selection effectiveness. We retain their unfavorable validation results and confidence intervals transparently. Cor. 5.7 is nevertheless a proved guarantee for a PL information objective; translating it into better learning outcomes is a separate empirical question.

Critical Weakness 2 — Finite-sample evidence. Appendix K's unrestricted 60-cell hypothesis failed: log–log slope −3.7497-3.7497, R2=0.3859R^2=0.3859, versus the predicted −1-1. Restricting to NγPL≥10N\gamma_{\rm PL}\ge10 was post hoc and does not provide a finite-sample theorem. The revised Appendix K qualifies the severe low-information errors rather than treating every case as a proven estimator failure. None of the main population-Fisher proofs depends on this regression. Appendix L.1 supplies a new audit of the unchanged data, not a replacement hypothesis.

Question 1 — Why choose the threshold, and how sensitive is the result? There is no independent theorem making 10 a universal cutoff. NγPLN\gamma_{\rm PL} measures accumulated information in the least-identifiable direction and motivates checking the regime in which MLE error follows inverse-Fisher scaling. Appendix L.1 reproduces both published fits to machine precision from the frozen 60-cell dataset, using one 200-replicate mean per cell. At thresholds t=8,10,12,15,20,30t=8,10,12,15,20,30, slopes range from −1.0045-1.0045 to −0.9922-0.9922, with R2≥0.9944R^2\ge0.9944. Thresholds 8 and 10 select the identical 51 cells; at 10, leave-one-cell-out slopes lie in [−1.0079,−1.0006][-1.0079,-1.0006].

The limitation is explicit: eight of nine cells with NγPL≤6.34N\gamma_{\rm PL}\le6.34 have anomalous errors, and retaining lower-information cells (t=2t=2–5) changes slopes to −1.206-1.206 through −1.142-1.142 (R2≈0.90R^2\approx0.90). For six severe cells, exact calculations predict approximately 10–181 of 200 replicate datasets with a separation event precluding a finite PL MLE. This supports severe estimation instability in some cases but does not explain every anomaly; per-replicate diagnostics were not saved. The cutoff remains post hoc, and no universal finite-sample scaling law is claimed.

Question 2 — What if reward scores are estimated? Appendix L.2, Prop. L.3, provides a conditional robustness result. Assume every within-group pairwise score-gap error is bounded by δ\delta. Since ∣∂Δlog⁡κlog⁡∣≤1/σ|\partial_\Delta\log\kappa_{\log}|\le1/\sigma, true and estimated reward-gap Laplacians differ by multiplicative factors e±δ/σe^{\pm\delta/\sigma}. If S^\hat S maximizes the estimated λ2(LBT)\lambda_2(L_{\rm BT}) and S⋆S^\star maximizes true PL curvature among fixed-size groups, then

γPL(rS^,σ)≥e−2δ/σK−1γPL(rS⋆,σ).\gamma_{\rm PL}(r_{\hat S},\sigma)\ge\frac{e^{-2\delta/\sigma}}{K-1}\gamma_{\rm PL}(r_{S^\star},\sigma).

The guarantee weakens when gap error is large relative to σ\sigma; it does not assert that real reward models satisfy the assumption. Appendix E's exact K=4K=4 example shows spectral selection choosing a group with 11.4×11.4\times greater curvature than trace selection, but this is not a learning experiment.

We ask that the mathematical soundness and significance of these proved results be assessed separately from the still-open question of downstream effectiveness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.