- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors compare three tie-handling rules in a ranking experiment with 12 candidate systems. The paper reports the same winner sequence under all three rules and a 0.09 spread in mean Kendall correlation with a stated ground-truth ranking. It also describes a ledger-based process for inserting experimental results into the manuscript.
Critical strengths
Negative results are being disclosed.
Critical weaknesses
The paper never properly explains what experiment was run. Moreover, the title contradicts the results, and even the counts in the supposedly audited reporting system do not agree. There is also no connection with recent literature.
Questions for authors
See above.
Author Response to Reviewer gX9E: Spec, Title Alignment & Ledger Counts
We thank Reviewer gX9E for the direct and incisive review. We have completely rewritten the paper and expanded our experiments to resolve all four criticisms:
1. Complete Experimental Specification (Section 3): We replaced the fragmented ledger dump with the full mathematical formulation: K=12 candidates with latent skills θ∈[−1,1]12 (
SYNTH_GT, Eq. 1), pairwise judge margins Dij(r)=(θi−θj)+ηij(r) (SIMULATED, Eq. 2), tie quantization ∣Dij(r)∣≤ϵ (Eq. 3), and scoring rulesDrop(Wi/(Wi+Li)),Split((Wi+0.5Ti)/Ni), andLoss(Wi/Ni).2. Reconciling the Title with the Baseline Results (Theorem 1 & Table 2):
Drop,Split, andLosswhenever the #1 vs. #2 win lead satisfies W(1)−W(2)≥21(T(2)−T(1)), while interior candidates with heterogeneous tie rates invert (Ti−Tj>2(Wj−Wi), yielding Δτ=0.0878).Split,Drop, andLossflips the top-1 leaderboard winner in 10.0% to 26.7% of trials, directly substantiating the title.3. Reconciled Provenance Ledger Counts (Appendix B, Table 5): In our initial draft, the text cited 153 raw JSONL lines (
71SYNTH_GT+82SIMULATED, which count per-seed array cells) while the appendix table listed 85 deduplicated summary keys (13SYNTH_GT+72SIMULATED). Both counters are now explicitly reconciled in Appendix B (Table 5).4. Connection to Recent Literature (24 Compiled References): Section 2 now connects our work to LLM-as-a-judge arenas (Zheng et al., 2023; Chiang et al., 2024; Dubois et al., 2024; Boubdir et al., ICLR 2024) and paired-comparison tie models (Bradley & Terry, 1952; Rao & Kupper, 1967; Davidson, 1970; Arrow, 1951; Young, 1974).