- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper considers a scenario in which a group of ranked items is given, and we seek to learn the underlying (true) scores that describe those preferences. For this, the paper studies the local Fisher information, which a single complete Plackett-Luce ranking of the K items provides about the true underlying scores. They base their analysis on the smallest (non-zero) eigenvalue of the Fisher information . They show that the Fisher matrix of the Plackett-Luce model is equivalent to a Laplacian, with edge weights being the Bradley-Terry discriminability times an exposure coefficient. Further, they obtain (Th. 5.5) a two-sided bound on , based on the reward-gap Laplacian algebraic connectivity of the Bradley Terry Fisher Information. Two main conclusions can be drawn: (A) For symmetric scores, increasing the group size K can only improve boundedly. (B) For non-symmetric scores, is restricted by the algebraic connectivity of the reward-gap graph between the underlying items. As such, scores that split into well-separated clusters are less beneficial than those that exhibit smaller between-cluster gaps.
Critical strengths
[S1] Theoretical Soundness: The theoretical derivations ( I have checked some but not all) seem sound and provide a detailed characterization of the problem. Furthermore, I find the factorization the strongest theoretical insight of the work.
[S2] Separability between trace: Central to this work, the authors convincingly show that the trace can convey insights different from those of (see Sec. 6.1). As such, it gives a basis why might be of practical benefit, yet this requires more in-depth and realistic evaluations (See [W1]).
Critical weaknesses
[W1] Lack of Practical Implication Investigation: With Corollary 5.7, a statement is provided on how to select different candidate groups of size K for practical applications, since computing the Bradley-Terry information is much cheaper than computing the complete Plackett-Luce Hessian. I find that such a statement should prompt at least small-scale experiments to demonstrate whether practical benefits can be gained from these theoretical insights. This would substantially benefit the paper.
[W2] Readability and Presentation: The paper is structured in largely self-contained sections, without a clear storyline connecting them. For example, I find the connection from symmetric to the asymmetric analysis hard to follow intuitively and would benefit from a more overarching storyline. The appendix (in particular Appendix H) seems to contain results from a previous submission. Table H.3 summarizes the results in Table 6 only qualitatively, whilst the latter is not referenced. In general, Appendix H is written to contain results as further evidence, yet H.4 and its Table 7 contain results noted as „secondary“ by the authors. The paragraph on „Prior validation code“, which probably suggests changes in the original code basis, is uninformative in the Appendix of a paper.
[W3] Related Work Comparison: As the authors have written, Hajek et al. already compared with the graph Laplacian and used the Cramer-Rao bound, which is closely related to the Fisher information. To my current understanding, this is conceptually close to the present work; therefore, I would appreciate a more detailed comparison of the differences between them. See [Q4].
Questions for authors
[Q1] How tight is the Spectral sandwich in Theorem 5.5? In particular, is the lower bound of tight, or could a tighter one be achieved?
[Q2] What would be the practical implications or recommendations based on Corollary 5.7?
[Q3] Given that the theoretical analysis is only restricted to a fixed complete ranking, how would the results change beyond that? Could you give insights here?
[Q4] Could you clarify to what extent your results differ from those of Hajek et al., as I currently understand that both papers deal with analyzing the Plackett Luce model via Fisher bounds based on the Laplacian.
3 comments
Thank you for your response, and I apologize for the late reply. I lost track of this.
I have read the rebuttal and the revised paper, and they have not convinced me. I therefore keep my original score and list the details of this decision below:
Empirical experiments [RW1]
There are still no experiments that show practical use. The new Appendix E is an exact calculation on two hand-picked groups. I would have wanted some real experiments demonstrating that the theoretical insights are meaningful to downstream users, or at least a rationale for why this is not necessary.
New storyline [RW2]
The changes are not marked (e.g., in blue), which makes it hard to follow what was revised. The paper has changed substantially, and I believe a version with this many changes needs a new round of review. I also don’t find it much easier to understand: the sections still read as largely separate pieces, and Appendix H still contains results from an earlier submission that you call secondary and that do not test the proposed criterion.
Benefit of the new insights [RW3]
It is still unclear what exactly the comparison with Hajek et al. is. The authors state the difference but do not show, formula by formula, what is new and what is a reformulation. In particular, how does Thm. 5.3 differ from Hajek et al.'s results? Appendix J adds no further insight here; you state yourselves that it only expands on §7.
Clarification of Theoretical Contributions, Revisions, and Prior-Work Comparison
We thank the reviewer for the follow-up and for taking the time to examine the revised manuscript. We address the three remaining concerns directly.
RW1 — Practical experiments.
The paper's contributions are theoretical: the exact arbitrary-score factorization of the Plackett–Luce Fisher operator (Theorem 5.3), universal exposure bounds (Theorem 5.4), a two-sided spectral comparison (Theorem 5.5), and a group-selection criterion with a provable approximation guarantee (Corollary 5.7).
These results characterize the information geometry of complete ranking observations. They do not establish that a particular LLM training algorithm benefits from the proposed selection criterion. Downstream evaluation remains an open empirical question, and we do not claim otherwise.
RW2 — Changes in the revision.
The research question and central results (Theorems 3.1, 4.1, and 5.3–5.5) are unchanged from the original submission. The revision adds equation numbering, main-text proof guidance, clearer attribution of prior results, and supporting appendices.
Appendix H was titled "Historical Supporting Experiments" in the original submission. It is retained for transparency and does not validate the proposed spectral selection criterion. We acknowledge that the revisions were not visually highlighted, making the changes harder to identify.
RW3 — Mathematical comparison with Hajek, Oh & Xu (2014).
The correspondence between the results is as follows:
| Result | Hajek et al. (2014) | This paper |
|---|---|---|
| Stagewise Fisher structure | Proof of Theorem 2 (§3.2, arXiv version) | Established result; credited in §3 |
| Equal-score Fisher curvature | Proof of Theorem 2 (§3.2, arXiv version) | Theorem 3.1; established result |
| BT discriminability × listwise exposure | Not isolated as a separate factor | Theorem 5.3, eq. (6) |
| Universal exposure bounds, | No corresponding bounds identified | Theorem 5.4, eq. (7) |
| Curvature comparison | Co-occurrence Laplacian (Definition 3); range-dependent eigenvalue bound (Lemma 2) | Score-dependent BT Laplacian; two-sided operator bound (Theorem 5.5, eq. (8)) |
| Fixed-range curvature separation and group-selection guarantee | No corresponding guarantees identified | Corollaries 5.6–5.7 |
The specific distinction in Theorem 5.3 is that the exact aggregate PL Fisher operator is expressed as
where depends only on the pairwise score gap, while captures exposure within the full candidate list.
Theorem 5.4 establishes universal bounds on this exposure coefficient. These yield the two-sided operator comparison in Theorem 5.5:
The multiplicative constants and are independent of score range. The comparison operator retains score dependence through the pairwise discriminability kernel .
Clarification of Novelty and New Spectral Group-Selection Results
We thank the reviewer for recognizing the exact factorization and the trace-versus-weakest-curvature distinction. The revision responds to each weakness and question explicitly.
Critical Weakness 1 — No demonstrated practical benefit. We agree that downstream gains from spectral group selection remain untested. Cor. 5.7 is a proved approximation for the PL Fisher-curvature objective, not a learning-performance claim. New Appendix E provides an exact K=4 two-group illustration: at R/σ=3, the spectral criterion chooses the group with 11.4 times the exact weakest-direction curvature of the trace-preferred group, even though the groups have equal size and range. The respective traces are 0.638/σ2 and 1.010/σ2. This demonstrates a decision consequence, not improved finite-sample estimation or LLM post-training.
Critical Weakness 2 — Storyline and Appendix H. The revised progression is: previously known symmetric baseline (Thm. 3.1) → proper-subset obstruction (Thm. 4.1) → arbitrary-score exposure factorization (Thm. 5.3) → spectral consequences (Thm. 5.5). We numbered equations (1)–(10) and added main-text proof guidance. Appendix H now states that the historical Pythia and regex-lite experiments used a trace-based, not spectral, selector; it includes unfavorable validation results and confidence intervals. Appendix I explains the prior validation-code distinction and its limits. Those historical results do not test Cor. 5.7, and the Discussion no longer presents them as validation of spectral selection.
Critical Weakness 3 — Comparison with Hajek, Oh & Xu (2014). We agree that the overlap is important. Their analysis of PL rankings on assigned subsets already gives pairwise stagewise Fisher terms and the equal-score calculation underlying our Thm. 3.1 (proof of their Thm. 2). The revised §3 explicitly credits this, and §7 and App. J sharpen the comparison. Their curvature comparison uses a score-independent co-occurrence Laplacian with constants depending on score range. Our contribution is not the generic Laplacian form: at arbitrary scores, Thm. 5.3 (6) separates each edge into BT discriminability and listwise exposure; Thm. 5.4 bounds exposure using the first-hit identity (7); and Thm. 5.5 (8) gives a score-dependent, range-free operator sandwich. Cor. 5.6 and 5.7 derive fixed-range separation and group selection from it. We do not claim novelty for the symmetric formula.
Question 1 — How tight is the sandwich? The universal 1/(K−1) coefficient need not be sharp. For K=3, eq. (9) gives αab=1−q+q2≥3/4, hence 43LBT⪯Hσ⪯LBT. This 3/4 is sharp: with r=(0,0,−t) and contrast v∝(−1/2,−1/2,1), the Rayleigh ratio approaches 3/4 as t→∞. The optimal general-K constant remains open; numerical slack alone does not settle it.
Question 2 — What should a practitioner do with Cor. 5.7? For a finite family of equal-size groups, obtain target or estimated scores, build LBT from pairwise gaps using (3), and select the group maximizing λ2(LBT) before collecting rankings. Cor. 5.7 guarantees at least 1/(K−1) of the best group's true PL curvature; for K=3, 3/4 applies. Appendix L.2 (Prop. L.3) shows the factor becomes e−2δ/σ/(K−1) when every within-group estimated gap error is bounded by δ. No practical reward-model accuracy is assumed without this condition.
Question 3 — Does the theory extend beyond one ranking? Yes, for fixed independent complete PL ranking observations on repeated or overlapping groups in a common score space. Appendix L.3 (Prop. L.4) proves additive Fisher information and
∑tKt−1Lt⪯Htotal⪯∑tLt.
The aggregate operators have the same kernel; identifiability depends on connectivity of the union graph. Adaptive selection requires a predictable history-based rule and is not an adaptive finite-sample guarantee.
Question 4 — Precisely how does this differ from Hajek et al.? The distinction is not whether PL Fisher is a Laplacian, nor the equal-score calculation: both precede us. It is the exact arbitrary-score edge factorization and comparison with a score-dependent BT reward-gap Laplacian, whose multiplicative constants are independent of reward range. That makes the listwise effect of third candidates explicit and supports Cor. 5.6–5.7. See revised §7 and Appendix J.
We hope the revised attribution and explicit limits clarify the novelty and significance of the proved results.