- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates non-local instability in LLM-based ranking systems, where modifying the description of one candidate can change the relative rankings of other candidates whose information remains unchanged. The authors propose a distribution-based evaluation framework using calibrated total variation distance and a non-local stability score to measure these effects while accounting for sampling variability. Experiments across product recommendation, resume ranking, and synthetic student ranking demonstrate that non-local instability occurs across multiple LLMs, even under attribute-preserving modifications. The paper further investigates the relationship between multi-attribute evaluation and ranking instability. The results suggest that reasoning can reduce sensitivity to local modifications, although it may increase variability across repeated rankings. Explicit evaluation criteria are also explored as a potential mitigation strategy.
Critical strengths
The main strength of this paper is its systematic investigation of non-local instability in LLM-based ranking. The proposed evaluation framework distinguishes changes caused by candidate modifications from ordinary sampling variability. The experiments cover multiple models, ranking tasks, and intervention types, providing useful evidence of the phenomenon. The additional analysis of reasoning and explicit evaluation criteria offers insights into potential approaches for improving ranking stability.
Critical weaknesses
Although the paper demonstrates non-local instability across multiple settings, several limitations affect the interpretation and generalizability of its findings. First, the evaluation includes only valid complete rankings, while some models exhibit substantial output failure rates. For example, Table 7 reports a 67.01% invalid-attempt rate for Gemma, raising concerns about whether the filtering and resampling procedures could influence the reported stability measurements. Second, the decoding configurations differ across models, making it difficult to attribute observed cross-model differences solely to model capabilities. Finally, the proposed mitigation through explicit evaluation criteria is primarily demonstrated on small-scale synthetic student-ranking tasks. Its effectiveness in realistic recommendation and hiring scenarios remains insufficiently established.
Questions for authors
- The stability evaluation considers only valid complete rankings, while some models have relatively high output failure rates. Could the authors clarify how invalid outputs and repeated sampling affect the reported stability results? Could the existing experimental records be used to compare failure rates between original and modified conditions?
- Different models are evaluated under different decoding configurations. Could the authors clarify how these differences might affect the cross-model stability comparisons and distinguish model-specific behavior from inference-configuration effects?
- Explicit evaluation criteria substantially improve ranking stability in the controlled student-ranking experiments. Could the authors further clarify the limitations of this mitigation strategy in more realistic recommendation and hiring scenarios? How should the existing results be interpreted when evaluation criteria cannot be fully specified?
2 comments
Thank you again for your thoughtful feedback and the time you have devoted to reviewing our work. We sincerely hope that our responses have sufficiently addressed your concerns. As the discussion deadline is now less than three days away, we would greatly appreciate any further feedback you might wish to share. Thank you again for your time and consideration!
Response to Reviewer avGF
We sincerely appreciate your detailed feedback and insightful questions. Please allow us to address your concerns comprehensively.
Q1. How failure rate is calculated and its influence.
Thank you for raising this important point. In Table 8, the reported 67.01% actually is the fraction of invalid attempts among all generation attempts, including retries, rather than an equally weighted average failure rate across inputs. Because we seek a fixed number of valid rankings per input, difficult inputs can contribute disproportionately many attempts. For example, if a certain input produces 30 invalid rankings out of 32 attempts and maintains this success probability, obtaining 32 valid rankings would require 512 attempts in expectation, compared with 32 for an input that always succeeds. Consequently, repeated failures on a small number of difficult inputs can substantially increase the aggregate attempt-level failure rate. The average failure probability of an input is much lower, for Gemma-4-E4B-it, it is around ~3%.
To compare Original and Modified conditions without this retry weighting, we conducted an additional fixed-attempt audit of Gemma‑4‑E4B‑it across shopping, hiring, and synthetic students tasks. We evaluated 60 matched O/M comparisons covering 3424 generations in total (without retry).
Detailed setting is in Appendix C. Note that the +1.30pp gap does not mean modification results in a higher failure rate. Actually two comparisons had higher failure rates under Original, two had higher rates under Modified. Thus, the observed differences were not uniformly in one direction.
Q2. Decoding configurations.
We agree that decoding settings can affect both within-condition variability and sensitivity to candidate edits. Our cross-model results therefore characterize each model under its evaluated inference configuration, rather than isolating differences in model capability. This is also why our formal definition treats the ranker as the model together with its decoding configuration.
Within every Original/Modified comparison, we hold the model, provider, and decoding settings fixed, changing only the target candidate's description. Thus, differences in decoding configurations across models do not confound the paired intervention comparison within each model. And we did not select any certain temperature, for the API models, we are using the providers' default temperature settings.
An interesting finding is that greedy decoding produces a substantially heavier tail in the survival curves, as shown in Appendix B (Figure 8). Because greedy decoding is unavailable for most API models, we evaluated this setting only on Qwen3-14B. Our results show that non-local instability remains present under greedy decoding.
Q3. Explicit-criteria mitigation in a more complex and realistic setting.
We added a shopping experiment in Appendix H.2 using DeepSeek‑V4‑Flash‑0731 on the same setting in Section 6.2 (Table 2) to evaluate mitigation beyond synthetic tasks with predefined attribute scores. Candidates are described through natural-language product information, without supplied evaluation scores.
Rather than prescribing numerical scores or a complete aggregation formula, we simply ask the model to consider price, past reviews, and performance and quality, prioritizing performance and quality while giving relatively little weight to price.
As shown above, an explicit-criteria clarification can reduce the consistency cost of reasoning while improving Non-local Stability Score. This also provides evidence that mitigation need not rely on synthetic attributes or a fully specified numerical rule.
When evaluation criteria cannot be fully specified, our practical insight is therefore to make the user's preference at least partly explicit. This does not guarantee the elimination of non-local instability or sampling variability: both remain in this experiment, and reasoning still increases dispersion relative to the corresponding Off setting (which is a limitation discussed in the end of Section 6.3). Nevertheless, the results support preference clarification as a practical way to partially mitigate these problems in a realistic recommendation task.
We hope this response clarify your questions and provide additional insights. Please feel free to reach out for any further discussion or clarifications!