acceptodds
Under review as a conference paper at ICLR 2027

One Candidate Changes the Rest: Non-Local Instability in LLM-based Ranking

Abstract

Large language models (LLMs) are increasingly used to recommend and rank candidates in a variety of settings, including e-commerce and hiring. In these LLM-as-a-Recommender scenarios, models are often tasked with mapping an unordered set of candidates to an ordered ranking. While prior work has shown that LLM-based rankers can be sensitive to presentation order, we identify a distinct and previously underexplored property of these rankers: Non-Local Instability. Ideally, a non-locally stable ranker should ensure that modifying a single candidate affects only that candidate's position relative to the others. However, we show that current autoregressive LLMs systematically violate this property: small, localized changes to the description of a single candidate can alter the relative ordering of other candidates whose descriptions remain entirely unchanged. We characterize this phenomenon across multiple types of perturbations, evaluate its prevalence across different models, and investigate its underlying mechanisms. Our experiments show that non-local instability is widespread and persists even in recent state-of-the-art models. Reasoning can substantially mitigate this effect, but at the cost of increased ranking variability under the same context. These findings reveal a fundamental robustness and fairness concern in LLM-based ranking systems: the relative ranking assigned to a candidate can be substantially influenced by seemingly irrelevant semantic changes to other candidates.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Evaluation, LLM-based Ranking, Ranking Robustness, Recommendation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_avGFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates non-local instability in LLM-based ranking systems, where modifying the description of one candidate can change the relative rankings of other candidates whose information remains unchanged. The authors propose a distribution-based evaluation framework using calibrated total variation distance and a non-local stability score to measure these effects while accounting for sampling variability. Experiments across product recommendation, resume ranking, and synthetic student ranking demonstrate that non-local instability occurs across multiple LLMs, even under attribute-preserving modifications. The paper further investigates the relationship between multi-attribute evaluation and ranking instability. The results suggest that reasoning can reduce sensitivity to local modifications, although it may increase variability across repeated rankings. Explicit evaluation criteria are also explored as a potential mitigation strategy.

Critical strengths

The main strength of this paper is its systematic investigation of non-local instability in LLM-based ranking. The proposed evaluation framework distinguishes changes caused by candidate modifications from ordinary sampling variability. The experiments cover multiple models, ranking tasks, and intervention types, providing useful evidence of the phenomenon. The additional analysis of reasoning and explicit evaluation criteria offers insights into potential approaches for improving ranking stability.

Critical weaknesses

Although the paper demonstrates non-local instability across multiple settings, several limitations affect the interpretation and generalizability of its findings. First, the evaluation includes only valid complete rankings, while some models exhibit substantial output failure rates. For example, Table 7 reports a 67.01% invalid-attempt rate for Gemma, raising concerns about whether the filtering and resampling procedures could influence the reported stability measurements. Second, the decoding configurations differ across models, making it difficult to attribute observed cross-model differences solely to model capabilities. Finally, the proposed mitigation through explicit evaluation criteria is primarily demonstrated on small-scale synthetic student-ranking tasks. Its effectiveness in realistic recommendation and hiring scenarios remains insufficiently established.

Questions for authors
  1. The stability evaluation considers only valid complete rankings, while some models have relatively high output failure rates. Could the authors clarify how invalid outputs and repeated sampling affect the reported stability results? Could the existing experimental records be used to compare failure rates between original and modified conditions?
  2. Different models are evaluated under different decoding configurations. Could the authors clarify how these differences might affect the cross-model stability comparisons and distinguish model-specific behavior from inference-configuration effects?
  3. Explicit evaluation criteria substantially improve ranking stability in the controlled student-ranking experiments. Could the authors further clarify the limitations of this mitigation strategy in more realistic recommendation and hiring scenarios? How should the existing results be interpreted when evaluation criteria cannot be fully specified?
2 comments
AuthorsOpenReview ↗

Response to Reviewer avGF

We sincerely appreciate your detailed feedback and insightful questions. Please allow us to address your concerns comprehensively.


Q1. How failure rate is calculated and its influence.

Thank you for raising this important point. In Table 8, the reported 67.01% actually is the fraction of invalid attempts among all generation attempts, including retries, rather than an equally weighted average failure rate across inputs. Because we seek a fixed number of valid rankings per input, difficult inputs can contribute disproportionately many attempts. For example, if a certain input produces 30 invalid rankings out of 32 attempts and maintains this success probability, obtaining 32 valid rankings would require 512 attempts in expectation, compared with 32 for an input that always succeeds. Consequently, repeated failures on a small number of difficult inputs can substantially increase the aggregate attempt-level failure rate. The average failure probability of an input is much lower, for Gemma-4-E4B-it, it is around ~3%.

To compare Original and Modified conditions without this retry weighting, we conducted an additional fixed-attempt audit of Gemma‑4‑E4B‑it across shopping, hiring, and synthetic students tasks. We evaluated 60 matched O/M comparisons covering 3424 generations in total (without retry).

Original failure rateModified failure rateM−O
2.76%4.06%+1.30 pp

Detailed setting is in Appendix C. Note that the +1.30pp gap does not mean modification results in a higher failure rate. Actually two comparisons had higher failure rates under Original, two had higher rates under Modified. Thus, the observed differences were not uniformly in one direction.


Q2. Decoding configurations.

We agree that decoding settings can affect both within-condition variability and sensitivity to candidate edits. Our cross-model results therefore characterize each model under its evaluated inference configuration, rather than isolating differences in model capability. This is also why our formal definition treats the ranker as the model together with its decoding configuration.

Within every Original/Modified comparison, we hold the model, provider, and decoding settings fixed, changing only the target candidate's description. Thus, differences in decoding configurations across models do not confound the paired intervention comparison within each model. And we did not select any certain temperature, for the API models, we are using the providers' default temperature settings.

An interesting finding is that greedy decoding produces a substantially heavier tail in the survival curves, as shown in Appendix B (Figure 8). Because greedy decoding is unavailable for most API models, we evaluated this setting only on Qwen3-14B. Our results show that non-local instability remains present under greedy decoding.


Q3. Explicit-criteria mitigation in a more complex and realistic setting.

We added a shopping experiment in Appendix H.2 using DeepSeek‑V4‑Flash‑0731 on the same setting in Section 6.2 (Table 2) to evaluate mitigation beyond synthetic tasks with predefined attribute scores. Candidates are described through natural-language product information, without supplied evaluation scores.

Rather than prescribing numerical scores or a complete aggregation formula, we simply ask the model to consider price, past reviews, and performance and quality, prioritizing performance and quality while giving relatively little weight to price.

InstructionReasoningCal.NLS ↑Kendall dispersion ↓
DefaultOff71.880.0477
DefaultOn92.250.0960
Clarified prioritiesOff82.890.0379
Clarified prioritiesOn93.910.0825

As shown above, an explicit-criteria clarification can reduce the consistency cost of reasoning while improving Non-local Stability Score. This also provides evidence that mitigation need not rely on synthetic attributes or a fully specified numerical rule.

When evaluation criteria cannot be fully specified, our practical insight is therefore to make the user's preference at least partly explicit. This does not guarantee the elimination of non-local instability or sampling variability: both remain in this experiment, and reasoning still increases dispersion relative to the corresponding Off setting (which is a limitation discussed in the end of Section 6.3). Nevertheless, the results support preference clarification as a practical way to partially mitigate these problems in a realistic recommendation task.


We hope this response clarify your questions and provide additional insights. Please feel free to reach out for any further discussion or clarifications!

Thank you again for your thoughtful feedback and the time you have devoted to reviewing our work. We sincerely hope that our responses have sufficiently addressed your concerns. As the discussion deadline is now less than three days away, we would greatly appreciate any further feedback you might wish to share. Thank you again for your time and consideration!

Reviewer_MFeNOpenReview ↗
Rating
2
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates non-local instability in LLM-based ranking. The authors identify and characterize a phenomenon in which modifying the description of a single candidate can change the relative rankings of other candidates whose descriptions remain unchanged. A distributional evaluation framework based on total variation (TV) and a Non-Local Stability (NLS) score is proposed to quantify this phenomenon while accounting for sampling variability. Experiments across product recommendation, resume ranking, and synthetic student ranking demonstrate that non-local instability occurs across different LLMs and can arise even from attribute-preserving modifications. The paper further examines the role of multi-attribute integration through attribute-alignment analyses and activation steering. Finally, the authors show that reasoning can reduce non-local instability but may increase ranking variability, while explicit evaluation criteria can mitigate this trade-off in controlled settings.

Critical strengths
  1. The paper investigates an interesting and practically relevant issue concerning the robustness and consistency of LLM-based ranking systems.
  2. The authors distinguish intervention-induced ranking changes from variability under identical inputs. The use of repeated sampling, target removal, and finite-sample calibration provides a more rigorous evaluation methodology.
  3. The experiments cover multiple LLMs, application domains, and perturbation types, providing evidence that non-local instability is a widespread phenomenon among autoregressive LLM rankers.
Critical weaknesses
  1. The experiments focus exclusively on ranking lists containing ten candidates, with relatively limited diversity in some application settings, which limits the generalizability of the findings. Evaluations with larger candidate sets, more diverse ranking domains, and different ranking formulations are needed to verify whether the findings generalize to realistic recommendation and retrieval scenarios.
  2. The paper lacks comparisons with alternative ranking strategies, making it difficult to determine whether non-local instability is primarily associated with joint listwise generation or represents a more general limitation of LLM-based ranking. Comparisons with pointwise, pairwise, and setwise ranking approaches are needed.
  3. The explicit-criteria mitigation is evaluated mainly on controlled synthetic student ranking tasks with predefined attribute aggregation rules. Its effectiveness in more complex and realistic settings remains unclear.
  4. The evaluation focuses primarily on ranking stability without considering ranking effectiveness. Additional evaluations using standard ranking metrics, such as Normalized Discounted Cumulative Gain (NDCG), are needed to determine whether improved stability also leads to better ranking performance or comes at the cost of ranking quality.
Questions for authors
  1. Why does reasoning reduce sensitivity to local candidate modifications while increasing ranking variability under identical inputs? Could the authors provide further insights into the factors contributing to this trade-off?

  2. How sensitive is the calibrated non-local instability score to the chosen sample size (m = 32 vs. 64), and would a larger sample budget alter the tail behavior of the survival curves?

3 comments
AuthorsOpenReview ↗

Response to Reviewer MFeN (Question Part)

We sincerely appreciate your detailed feedback and insightful questions . Please allow us to answer your questions first.


Q1. Why does reasoning reduce sensitivity to local modifications while increasing ranking variability under identical inputs?

In our paper, we suggest that robustness to an input intervention and consistency across repeated generations are distinct properties. And for further insights, here are three potentially contributing factors:

  • Why reasoning reduce sensitivity

① More explicit and independent candidate evaluation. In the reasoning traces, models often examine candidates individually first and compare them pairwisely. This may help separate candidate-specific evidence and reduce sensitivity.

② Reduced influence of small edits due to long reasoning text. A local modification may affect how the model interprets evidence or prioritizes evaluation criteria. Subsequent reasoning may reconsider these judgments and reduce the modification's influence. One possible explanation is that a small textual change becomes less influential within a much longer evaluation trajectory.

  • Why increase ranking variability

① Variation in attribute trade-offs across reasoning runs. Different attribute preferences can lead to markedly different rankings, even for identical candidate information. Reasoning may increase the diversity of these preferences because it introduces a much longer sequence of autoregressively sampled intermediate tokens before the final ranking. Our synthetic-student case study in Appendix I illustrates this possibility: one strategy strongly penalizes low GPA, whereas another allows strengths in other attributes to compensate, which cause huge disagreement in the ranking results.


Q2. Sample size.

Our additional experiment supports that m = 32 is an already sufficient sample size for the main conclusions. We add a paragraph in Appendix B, experiment details can be found there.

mMean Cal.NLS ↑Cal.NLS varianceTail TV p90Cal.TV > 0.5 Fraction
475.1643.45460.4388.827%
880.2721.73210.3534.373%
1682.1290.90770.3103.000%
3282.5130.46390.3022.727%
6482.2880.24650.3042.698%
12881.8530.12830.3102.754%

These results show that very small sample budgets (like 4 and 8) can materially affect both the score and the apparent tail, but doubling the budget from 32 to 64 does not materially change the tail summaries or remove the large-shift cases in this experiment. Due to computation budget limit, we consider 32 and 64 as practical sample size in our paper.


We hope this response clarify your questions and provide additional insights. Please feel free to reach out for any further discussion or clarifications!

AuthorsOpenReview ↗

Response to Reviewer MFeN (Weakness Part)

Here we want to address some concerns in critical weakness section.


W1. Experiment scale.

Our main evaluation has already covered two practically important applications: shopping recommendations and hiring. The main shopping panel contains 120 queries across five product categories. The hiring evaluation uses five vacancies with ten de-identified resumes each, three candidate orders, two targets, and two edits per target.

To broaden the evaluation beyond these applications, we added SST and FED experiments in Appendix J, extending the analysis to ranking natural-language movie reviews and complete conversations without supplied attribute scores.

We agree that larger candidate sets are an important extension. Given the available experimental budget, we now explicitly acknowledge this restriction in the limitations in Section 8. Thank you for helping us clarify this scope.


W2. Limitations on ranking strategies comparison.

Our primary focus is joint listwise generation, where all candidates appear in the same context and the model directly produces a ranking. This setting captures the most common recommendation interaction: the model considers the available alternatives together and recommends them in order.

With fixed scoring instructions and candidate-independent tie-breaking, in pointwise ranking, editing candidate C cannot change the score distributions of A or B when C never enters their scoring contexts. Pairwise evaluation similarly isolates the direct A-versus-B comparison from C.

Setwise methods introduce additional choices concerning subset construction and ranking aggregation, so we identify broader strategy comparisons as future work.


W3. Explicit-criteria mitigation in a more complex and realistic setting.

We added a shopping experiment in Appendix H.2 using DeepSeek‑V4‑Flash‑0731 on the same setting in Section 6.2 (Table 2).

InstructionReasoningCal.NLS ↑Kendall dispersion ↓
DefaultOff71.880.0477
DefaultOn92.250.0960
Clarified prioritiesOff82.890.0379
Clarified prioritiesOn93.910.0825

As shown above, an explicit-criteria clarification can reduce the consistency cost of reasoning while improving Non-local Stability Score. This also provides evidence that mitigation do not rely on synthetic attributes or a fully specified numerical rule.


W4. Ranking effectiveness.

We agree that stability should not be treated as a substitute for ranking quality. Since the shopping and hiring dataset do not contain labels that can be used to calculate NDCG, we therefore added NDCG@10 evaluations using SST's sentiment labels and FED's released human overall-quality ratings. Full results and experiment details can be found in Appendix J.1.

TaskSettingCal.NLS ↑NDCG@10 ↑
SSTw/o reasoning81.850.9409
SSTCriteria + reasoning89.390.9293
FEDw/o reasoning76.740.9526
FEDCriteria + reasoning90.660.9544

As shown above, mitigation of non-local instability generally does not harm ranking effectiveness, nor improves the quality. Stability and effectiveness are two separate properties, and our results motivate reporting them jointly in futher LLM-based ranking work.


Again we hope this response address your concerns and provide additional insights. Please feel free to reach out for any further discussion or clarifications!

Thank you again for your thoughtful feedback and the time you have devoted to reviewing our work. We sincerely hope that our responses have sufficiently addressed your concerns. As the discussion deadline is now less than three days away, we would greatly appreciate any further feedback you might wish to share. Thank you again for your time and consideration!

Reviewer_fr3uOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates a phenomenon in ranking by LLM-as-a-judge, which says that when a single item is altered, the entire ordering of the items could be disrupted. The paper further attempts to answer why should such instability emerge and then to provide a remedy.

Critical strengths

The paper discovers an interesting phenomenon when using LLMs to rank a group of items.

Critical weaknesses
  1. The insight that "More attributes lead to higher instability" is not enough / even problematic. An LLM does not really distinguish what is an attribute and what is the prompt formatting text. What if there is no explicitly formatted attribute at all from the candidates, e.g. answers for an open-ended question? Using the theory of the paper, the output ranking should be perfectly stable. But I think this is hard to be true.

  2. While this "instability" seems another important issue, the paper does not provide a principled way to solve it. The proposed remedy can be summarized into one heuristic "Let's turn on the reasoning mode.". While effective, the method relies on the increased reasoning capability itself, without further help. Hence, there is little methodological contribution.

Questions for authors

See the weaknesses above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer fr3u

Thank you for your feedback. We think there might be some misunderstanding of our paper, so here we want to clarify some points. Hope this might address your concern.

1. Our explanation does not require attributes to be explicitly formatted, and our conclusion still holds.

We distinguish the format of candidate evidence from the criteria used to evaluate it. An LLM can interpret and integrate evaluation-relevant information from natural-language text without receiving explicit attribute–value fields. In Section 5.1, we use synthetic student experiments to make measurement more tractable and rigorous, rather than because our explanation requires explicitly formatted attributes.

To clarify this point, we added experiments in Appendix J using SST, where movie reviews are ranked solely by expressed sentiment, and FED, where complete conversations are ranked using ten stated dialogue-quality criteria. In both tasks, candidates consist of natural-language text without supplied attribute scores. We use GPT-5.6 Luna; full experimental details are provided in Appendix J.

TaskRaw TVNull TVCal. NLS ↑
SST0.13030.052681.85
FED0.17120.068076.74

Both tasks exhibit non-local distribution shifts after removing the edited candidate. The simpler SST task, which involves fewer potential evaluation criteria, also exhibits greater stability in this comparison.

Thank you for prompting us to clarify this claim. This feedback has helped us improve the paper and communicate our findings more clearly and comprehensively.

2. LLMs do distinguish what is an attribute and what is the prompt formatting text.

Prior mechanistic work provides evidence that LLMs can represent semantic information and bind these attributes to entities, rather than treating all prompt content as same undifferentiated text:

  • Opiełka et al. [1] identifies concept representations that generalize across formats alongside format-dependent function vectors.
  • Feng and Steinhardt [2] identifies causal binding-ID vectors linking entities to attributes.
  • Dai et al. [3] shows that interventions on an ordering subspace change these associations.

Together, these findings can support that LLMs have the ability to distinguish what is an attribute.

3. Our mitigation strategy is not merely “turn on reasoning”; reasoning has its own drawback.

Our paper identifies both a benefit and a cost of reasoning: although it reduces sensitivity to local edits, it can increase within-condition dispersion, meaning that repeated runs under an identical input produce less consistent rankings. These are distinct forms of instability, so improved non-local stability alone does not resolve the reliability problem.

We therefore evaluate explicit evaluation rules combined with reasoning, rather than reasoning alone. Sections 6.2 (Table 2) and 6.3 (Table 3), together with Appendix H, document this trade-off and evaluate the combined mitigation on the synthetic-student task.

Reasoning alone can increase dispersion, whereas combining reasoning with an explicit rule reduces it below the baseline. The simplicity of this mitigation should not be considered a weakness. If a low-cost intervention can largely address a concerning phenomenon, a more complex modification—such as changing the training pipeline—would need to justify its substantially greater cost through additional benefits.

4. Why this problem matters and what our contributions are.

Non-local instability complicates description optimization: a merchant’s beneficial edit may lose its advantage or become harmful when competitors also edit their descriptions. It also raises fairness concerns in hiring, where one applicant’s wording can affect unchanged applicants. As discussed in the paper, candidates could potentially exploit this sensitivity to shift evaluation toward attributes favoring themselves.

Our contributions therefore extend beyond a mitigation heuristic: (i) phenomenon: characterizing non-local ranking shifts across tasks and models; (ii) mechanism: investigating attribute integration and tracing and controlling effects through representation interventions; and (iii) reasoning: identifying its stability–consistency trade-off and evaluating explicit rules to mitigate the consistency cost.

We hope this response clarify our claim and contribution in our paper. Feel free to reach out for any further clarifications!

[1] Opiełka et al. Causality ≠ Invariance: Function and Concept Vectors in LLMs. ICLR, 2026.

[2] Feng and Steinhardt. How do Language Models Bind Entities in Context? ICLR, 2024.

[3] Dai et al. Representational Analysis of Binding in Language Models. EMNLP, 2024.

Thank you again for your feedback. We sincerely hope that our responses have sufficiently addressed your concerns. As the discussion deadline is now less than three days away, we would greatly appreciate any further feedback you might wish to share. Thank you again for your consideration!

Thank you for your response. My concern is partly addressed. However, I still think the paper is not coherent in that the three phases: phenomenon finding, mechanism proposing, and methodology are not well connected. Non-local ranking effects, while important, is only one of the many inconsistencies that LLM has. What is important is to explain in depth (e.g. theory, mathematical formulation) why should such phenomenon appear, and then resolve / remedy the problem under this framework. These all the things the paper fundamentally lacks. I encourage the authors to further investigate such problems in a more principled way.

Thank you for your feedback! We are willing to clarify these points.

The coherence between the three phases is just as what you mentioned:

We first find an important and novel inconsistency in LLM-based ranking (to the best of our knowledge, this is the first research that points out non-local instability exists in various realistic scenrios) ->

Then we explain this phenomenon in depth with a mechanism interpretability perspective (especially showing that we can extract attribute preference steering vector from sementic-perserving changes, which indicates that LLM's perference does change due to these local minor modifications) ->

And then we give the problem a practical resolution (reasoning with explicit preference or rules). This realistic and easy-to-use method exactly comes from the insights in the mechanism interpretability analysis above.


About mathematical theory:

Most mathematical or statistics research on ranking strongly relys on a deterministic pointwise scorer, while generally people do not use LLMs like that when it comes to real life scenrios like shopping recommendations (which is exactly the setting in our research).

Also, this work mainly focuses on LLM evaluation, so mathematical analysis on ranking effectiveness (like in strongly mathematical related papers, such as Conformal ordering: Distribution-free predictive inference for orders and ranks [1]) is out of our scope and might be an entirely new research. We now add this suggestion into our Future Work section. Thank you!

Hope this help clarify our claim and scope.

[1] Yonghoon Lee and Zhimei Ren. Conformal ordering: Distribution-free predictive inference for orders and ranks. https://arxiv.org/abs/2610.40340

Reviewer_MaRnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates non-local instability in LLM-based ranking: editing one candidate can change the relative ranking distributions of candidates whose descriptions remain unchanged. The authors hold the query, candidate set, and presentation order fixed, remove the edited candidate before analysis, and compare distributions estimated from repeated ranking outputs. Experiments cover shopping, hiring, and synthetic student-ranking tasks across several models. The paper further examines changes in attribute alignment, transfers edit-derived representation directions to held-out candidate sets, and studies reasoning as a mitigation. Reasoning reduces edit sensitivity but can increase within-input ranking variability; explicitly specifying the evaluation rule improves both properties in controlled tasks.

Critical strengths

1、The paper studies a meaningful reliability problem using a carefully defined intervention. Holding the candidate set and order fixed, editing only one candidate, and removing that candidate before analysis help distinguish non-local changes from position effects and mechanical rank displacement. Repeated sampling further avoids treating a difference between two individual outputs as sufficient evidence of an intervention effect. This makes the phenomenon substantially more convincing than isolated examples of ranking reversals.

2、The analysis goes beyond documenting instability and identifies useful distinctions and mechanistic clues. The paper separates cross-condition sensitivity from within-condition consistency, showing that reasoning can improve one while worsening the other. The synthetic attribute controls, explicit-rule experiments, and transfer of edit-derived representation directions provide complementary evidence about possible sources of the effect. Although these do not establish a complete mechanism, they make the work informative and suggest productive directions for further study.

Critical weaknesses

1、The mechanistic account remains incomplete and does not sufficiently distinguish competing explanations. The results are consistent with local edits changing how the model integrates multiple attributes. However, changes in attribute alignment do not uniquely identify changes in attribute weights. Similar observations could arise from altered evidence interpretation, context-dependent normalization, near-ties, or dependencies introduced during autoregressive ranking generation. The representation interventions strengthen the evidence that edit-associated components can influence behavior on other candidate sets, but they do not establish that these components mediate the natural edit effect. A more systematic separation of these explanations would produce sharper conclusions. I view this as a limitation on the depth of the analysis rather than a reason to dismiss the empirical phenomenon.

2、The relationship between reasoning, repeated-output variability, and non-local sensitivity needs a more integrated explanation. The paper convincingly distinguishes these quantities, but it leaves open whether their changes share an underlying cause. One possibility is that reasoning explores different evaluation plans or attribute trade-offs across runs. Another is that broader output distributions partly reduce the measured separation between original and modified conditions. The current observations do not fully distinguish these interpretations. The explicit-rule experiments are useful, but a deterministic aggregation rule substantially simplifies the original judgment problem. It therefore remains unclear how the finding extends to realistic tasks with legitimate, underspecified trade-offs. Connecting the two forms of instability through a controlled intermediate decision process would substantially strengthen the analysis.

Questions for authors

1、Can the attribute-integration interpretation be separated from alternative mechanisms through a focused control? For example, fixing the attribute interpretation and aggregation rule while varying the edited candidate could help distinguish changes in evaluation criteria from execution or output-generation effects. Please clarify which existing evidence most directly supports a change in attribute trade-offs, rather than these alternatives.

2、What evidence connects reasoning-induced variability to changes in evaluation plans or attribute integration across runs? In particular, how should we distinguish a genuinely more edit-invariant decision process from reduced distributional separation accompanied by greater dispersion? A focused analysis linking run-level attribute preferences to ranking variation would help clarify the interpretation.

2 comments
AuthorsOpenReview ↗

Response to Reviewer MaRn

We sincerely appreciate your detailed feedback and insightful questions about attribute-integration interpretation. Please allow us to address your concerns comprehensively.

Q1. Can the attribute-integration interpretation be separated from alternative mechanisms (① execution problem, ② near-ties, ③ altered evidence interpretation)?

① Not entirely. Attribute integration and execution are difficult to separate because any evaluation procedure depends on reliable execution. In Appendix H (Table 14), we conduct the proposed control on the synthetic-student task: we explicitly specify attribute normalization and the aggregation rule while varying the edited candidate on the synthetic student task. The results are summarized below:

EvaluationReasoningCal. NLS ↑K_rule ↓
HolisticOff76.530.1526
HolisticOn96.000.0320
Explicit RuleOff79.740.1925
Explicit RuleOn100.000.0000

K_rule is the average normalized Kendall distance between the generated ranking and the ranking prescribed by the fixed evaluation rule. For holistic evaluation, it measures disagreement with that reference rule rather than an instruction violation.

Explicit rules alone do not guarantee reliable execution: without reasoning, the Explicit Rule condition still exhibits substantial rule disagreement and only a modest improvement in stability. These results show that specifying a rule and executing it reliably are distinct requirements. We therefore cannot fully disentangle attribute integration from execution or output-generation effects.

② Yes. Near-ties cannot explain the observed non-local instability.

We added a controlled experiment in which students A and B initially have identical attribute scores. Keeping B fixed (GPA 2.20, Peer 4.6, Social 4.6), we progressively increase A’s GPA and peer-evaluation scores while decreasing its social-contribution score, preserving their equal normalized average. At each level, we rename only a third student C and measure the resulting change in P(A > B).

As shown below, neither the raw nor the calibrated changes increase monotonically as the attribute profiles become more similar.

A GPAA PeerA SocialMean absolute A/B preference shiftMean calibrated pair TV
2.304.94.05.27 pp0.0404
2.405.23.40.98 pp0.0030
2.505.52.81.37 pp0.0025
2.605.82.22.93 pp0.0144
2.706.11.64.10 pp0.0256

These results rule out the simple explanation that closer attribute scores make untouched candidates more susceptible to non-local changes.

③ Yes. We added a shopping-ranking experiment in which candidates are described through textual evidence, without predefined evaluation scores. The model assigns each item scores for price, reviews, functionality, and honesty (the apparent credibility of its claims), and is ask to rank the candidates based on these evaluations. We then compare both the score distributions and the rank distributions before and after modifying one candidate’s description.

ReasoningScore Cal. TVRank Cal. TV
Off0.02730.0411
On0.04130.0303

As shown above, greater changes in score distributions do not necessarily translate into greater changes in rank distributions. Our hypothesis is that, across repeated runs, the relative scale used to interpret the evidence remains sufficiently stable, even when individual scores vary.

Q2. A focused analysis linking run-level attribute preferences to ranking variation

We have added a case study in Appendix I comparing two complete reasoning traces generated from the same synthetic-student input to provide some illustrative evidence. In one run, the model strongly penalizes a student’s low GPA despite strengths in the other four attributes; in the other, it explicitly allows those strengths to compensate for the lower GPA. The student consequently moves from eighth to first place, while a student with a higher GPA but weaker remaining attributes moves from third to seventh. These contrasting trade-offs suggest that run-to-run variation in attribute preferences may contribute to greater within-condition dispersion under reasoning.

We hope this response clarify your questions and provide additional insights. Please feel free to reach out for any further discussion or clarifications!

Thank you again for your thoughtful feedback and the time you have devoted to reviewing our work! Again, we sincerely hope that our responses have sufficiently addressed your concerns. As the discussion deadline is now less than three days away, we would greatly appreciate any further question or any further insight you might want to share and discuss. Thank you again for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.