acceptodds
Under review as a conference paper at ICLR 2027

LLMs Contain Multitudes: How Deployment Context Reshapes Model Preferences & Values

Abstract

The preferences and values attributed to large language models (LLMs) shape judgements about their bias, alignment and suitability for deployment. Recent pairwise evaluations increasingly ascribe stable, model-level preference and value systems to LLMs. However, accompanying robustness checks are limited to incidental prompt perturbations such as syntax variation and option reordering. They do not test whether the measured properties survive when the surrounding task context changes, as is common in real deployments. An audit may therefore provide false reassurance about the biases most consequential for a model's intended use. We test this context dependence directly using two established pairwise paradigms: country preference ranking and utility elicitation. In both, we make *deployment context* – the high-level task the model performs while making concrete value-dependent choices – our controlled variable. Across five LLMs and over 1.2M pairwise elicitations, we vary deployment contexts across framings such as writing a Reddit post or a news article. In preference rankings over 15 countries, context induces widespread, statistically significant rank shifts; each model's bias shifts systematically across contexts. Well-accepted patterns in prior work, such as Global North favouritism, also appear context-dependent. In utility elicitation over 50 outcomes, broad ordering remains consistent, but rankings within domains vary substantially, and cardinal utility ratios between outcomes (e.g. the utility of averting a death relative to receiving $1M) shift by at the median. Reported model-level preferences and utilities are therefore better understood as context-conditioned measurements than fixed model-level properties: bias and alignment conclusions drawn under one framing need not generalise to another.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, LLM evaluation, AI safety, AI alignment, bias, context sensitivity, preference elicitation, utility elicitation, model values

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_26wvOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper tests whether preferences elicited from language models change across neutral, news, Reddit, school, and vlog contexts. It studies country-preference rankings and pairwise preferences over outcomes used to fit utility models. Across five models and extensive repeated comparisons, context changes both local preferences and some inferred trade-offs, even when broad rankings remian comparatively stable. Additional analyses examine prompt wording, temperature, answer-only elicitation, explicit requests for the model's own preference, and the sampling-noise floor of utility-ratio variation.

Critical strengths
  1. The study examines a meaningful source of variation using complementary measurements. Moving beyond surface paraphrases to intended communication contexts is useful for understanding how preference assessments travel across applications. The distinction between stable overall ordering and unstable local comparisons or fitted trade-offs provides more information than an average rank correlation alone.

  2. The appendix contains controls that materially strengthen the result. Context effects persist in answer-only settings, and a country-preference control explicitly asks for preferences independent of audience or format. The utility study also compares ratio variation against a refitted permutation-based noise floor. These analyses make it less plausible that the findings are explained entirely by generated reasoning, one wording choice, or fitting noise.

Critical weaknesses
  1. The decision-level estimand is conditional on order consistency. The country-preference tests retain only AB/BA-consistent decisions, and retention rates differ across contexts. Consequently, these tests concern preferences conditional on consistency instead of preferences across all valid counterbalanced responses. I recognise that complete inconsistent pairs contribute zero to summed win-loss scores, so their exclusion need not change the country rankings themselves. The remaining concern is whether the reported decision-level significance and normalized effect sizes remain representative of the unfiltered responses.

  2. The mechanism and scope of the context effects remain incompletely identified. The main conclusion about context-conditioned elicited preferences is supported. However, the own-preference control in Appendix A.11 covers only one model in the country-preference task. It remains unclear whether the utility findings similarly persist when the evaluation objective is explicitly held fixed across framings. This limits interpretation of the utility shifts as changes in a model-level value system.

Questions for authors
  1. Please clarify which reported statistics are affected by excluding AB/BA-inconsistent pairs. In particular, do the decision-level context effects persist in an analysis of all valid counterbalanced responses that preserves the AB/BA pairing? For summed win-loss rankings, please confirm whether filtering leaves the rankings unchanged.

  2. Does an explicit fixed-objective/own-preference instruction preserve the key utility findings, or should that part of the conclusion be limited to context-conditioned communication tasks?

Reviewer_efAxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tests whether LLM preferences change when the same choices appear in different writing tasks. These tasks use neutral, news, Reddit, school-essay, and vlog framings. Five models compare every pair among 15 countries on six traits and among 50 outcomes across six domains. The paper uses a Thurstone-Mosteller model to infer numerical preference scores, called utilities, from choices between pairs. They fit separate scores for each context. The paper presents additional tests vary wording, sampling temperature, whether models explain their answers, and whether prompts explicitly request the model's own preference.

The paper reports changes in country preferences and geographic ranking gaps. It also examines changes in individual outcome rankings and ratios between preference scores, alongside stable overall rankings.

Critical strengths
  1. The paper presents well-designed studies to verify that changes in preferences persist under tests of alternative explanations. Table 2 demonstrates significant differences in country comparisons. Figure 3 shows a common direction of change. Some agreements are also presented agreement which could matter for alignment. For example, table 5 shows that sensitivity within a domain differs across models and domains, including some completely stable rankings.

  2. The study tests whether framing effects extend beyond one model or one kind of choice. This breadth makes the study useful for evaluating preference measurements across these tested settings. The authors also report releasing responses, extracted choices, fitted scores, and code. These materials could support independent checks and further analyses.

Critical weaknesses
  1. Ratios between fitted scores need clearer definitions and justification. The paper needs to distinguish changes in its fitted numbers from changes in how strongly a model values one outcome over another. This matters because scores near zero can also produce large changes when the calculation divides the largest ratio by the smallest. Using absolute ratios hides whether an outcome changes from "better than no change" to "worse than no change", or vice versa.

  2. Excluding choices that change with option order may affect the country findings. Section 4 keeps a trial only when the model chooses the same country after the two options reverse order. The selection itself changes with task framing, leaving unclear how faithfully these results describe all responses. Analyzing all valid choices while accounting for option order would determine whether the main ranking and North/South-gap conclusions survive this exclusion rule. Current evidence most directly supports context dependence among choices that stay consistent when option order reverses.

  3. Expressed choices in the proposed study do not equal to intrinsic values or behavior in all environments. The experiments show how models answer comparison prompts. They do not necessarily imply what values models hold internally or how models act when decisions have consequences. This distinction limits what the results imply for alignment beyond the tested measurements.

    • The tests in e.g. Figure 2 of expressed choices are static tests. Requesting the model's own preference does not independently establish that its answer measures an intrinsic value. Likewise, the writing-task framing does not directly test actual consequential behavior in deployment.
Questions for authors
  1. How does Table 8 handle pairs involving the zero-score outcome, other zero or near-zero scores, and scores that change sign?
  2. Do the main ratio changes persist when you report uncertainty in scores used as divisors and show that predicted choice probabilities adequately match observed choices?
  3. Please analyze all valid raw choices while accounting for option order. Do the main ranking changes and North/South-gap patterns persist without excluding choices that change when option order reverses? Please distinguish failures to extract an answer from disagreements after reversing option order, for each context. This would resolve whether context-dependent selection materially changes the main geographic conclusions.
3 comments
AuthorsOpenReview ↗

Official Comment by Authors (PART 1/2)

We thank the reviewer for the careful review. Below we conduct an additional analysis of our data to answer the critical weaknesses [W1,2,3] and questions [Q1,2,3]. Due to the amount of questions raised, we split our answer into two parts to properly answer them and detail how we will address them in our paper.

[W1]

We define our ratios by anchoring them to the “no change” outcome, set at 0. The absolute ratio captures changes in inter-outcome utility even when the ordering stays the same, and is unaffected by a common rescaling of the scores within each context. This does mean the ratio measures variation in relative utility under the fitted model, rather than direct preference strength. We will use the extra space in the camera-ready version to further clarify this.

We address W1’s further concerns in Q1 and Q2:

[Q1]

We report the median over the mean to reduce the influence of outlier ratios, although this does not completely negate sensitivity to near-zero scores. Removing pairs where either outcome’s average |normalised score|<0.1 in any context excludes about 25% of pairs. The median ratio shift changes from 2.47× to 1.99×, which is expected given the large number of pairs with above-average ratios removed. No zero-score outcomes were used.

We then focus on pairs whose scores change sign. In the paper, these pairs were kept to preserve information on magnitude shifts. Nevertheless, we agree that absolute ratios conceal changes in preference directions. Thus, the cleanest ratio shift value drops them. Removing sign-changing pairs shifts the median ratio shift to 2.01×, or 1.90× after applying the 0.1 threshold.

All of this shows that the reported trend is not confined to near-zero or sign-changing pairs, with the ratio shift still being significantly above the noise floor. We will add these subset results to the paper to clarify their influence.

[Q2]

To address uncertainty, we ran a bootstrapped analysis. For 1,000 resamples, we redrew with replacement the ten repeated answers per outcome pair per context, refitting utilities and recomputing ratios. Under our existing fitting procedure, the median ratio shift has a 95% bootstrap range of 2.34-2.67×, or 1.86-2.02× after excluding sign-changing pairs and pairs below the 0.1 threshold. The aggregate finding is therefore stable to resampling, with the median preventing sizeable impact of outliers. Furthermore, shuffling context labels removes their original association with the answers, reducing the median shift to 1.32× (1.26× with exclusions). This suggests that the observed shift reflects context dependence and is unlikely to be explained by random sampling alone.

We also checked whether the fitted utilities and noise scales predict observed choice frequencies. Under five-fold cross-validation, the mean absolute discrepancy across five confidence bands evaluated separately for each model was 1.3 percentage points. This close match supports using the fitted model to describe expressed preferences. We will add these results to the paper.

[W2]

We used AB/BA agreement to minimise in ordering bias elicited preferences, requiring the same country to be selected in both conditions. We agree this makes decision-level analyses conditional on order consistency. For the rankings themselves, however, complete AB/BA disagreement contributes one win and one loss to each country (i.e. a net zero). As such, country rankings and North/South gaps remain identical irrespective of inconsistent pairs being included.

For decision-level significance, retaining all complete AB/BA pairs (including disagreements) yields significant context differences in 90.7% of the 300 model-trait-context-pair comparisons. Originally this was 73.7%, using the same chi statistic and Holm-Bonferroni correction while preserving AB/BA pairing during permutation. Thus, not conditioning on agreement increases the proportion of significant context comparisons, reinforcing that context dependence does not rely on agreement filtering. As such, we retain agreement-only testing as the more conservative option.

[Q3]

Most of this question is covered in our answer to [W2]. For [Q3], we also investigate valid answers from incomplete pairs. First, we investigate their presence; extraction-failure / AB/BA-disagreement rates in the paper are:

Neutral: 0.16% / 17.83%

News: 0.49% / 21.73%

Reddit: 0.39% / 26.91%

School essay: 0.27% / 21.74%

Vlog: 0.87% / 21.21%

We examined valid responses from incomplete pairs by weighting AB and BA choice rates equally. With weighting, the results are nearly unchanged. Aggregate country ranks change by only 0.02 positions on average, and North/South gaps by 0.007 positions on average (maximum 0.10). All five models retain the subjective neutral-to-vlog shift toward the Global South.

Overall, this means that the main ranking and geographic bias conclusions remain robust across the tested data-filtering choices.

AuthorsOpenReview ↗

Official Comment by Authors (PART 2/2)

[W3]

In our work, we followed terminology used in previous papers, which proposed elicited preferences as evidence of internal value/bias structures [1,2]. One of our targets was to address these influential papers and to note the critical issue of context-dependence within them; this restricts us to the pair-wise choice experiment format. Nonetheless, we agree this format has its limitations.

Our ablations (Appendices A.10; B.3) show context dependence survives substantial changes to elicitation. For country-preferences in Llama-3.3-70B-Instruct, context differences remain significant in 60/60 comparisons under answer-only elicitation and 38/60 when explicitly requesting the model’s own preference, both after Holm-Bonferroni correction. Therefore, context dependence does not require generating the requested writing task or explanation, nor do own-preference instructions induce context-invariant preference profiles.

These checks narrow possible explanations for our observed context dependence. However, we agree they do not independently establish change in intrinsic values themselves, nor that elicited preferences predict consequential actions. We will make this distinction even more explicit throughout the paper.

[1] Kerche et al. (2026). [The silicon gaze]

[2] Mazeika et al. (2025). [Utility engineering]

Area_Chair_KSk6OpenReview ↗

take part in the discussion

Dear reviewer efAx,

pleaes take part in the discussion. time is running out.

thank you your ac

Reviewer_7KqUOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how deployment contexts affect LLM preferences and values. Through large-scale pairwise evaluations across five models, it shows that country preferences and utility rankings vary under different task framings, challenging the assumption that LLMs possess stable, context-independent preferences.

Critical strengths
  1. The paper investigates an issue in LLM evaluation: whether preferences and values measured under one context generalize to other deployment settings. This question is particularly relevant to model alignment and bias evaluation.
  2. The authors conduct over 1.2 million pairwise evaluations across five LLMs and multiple deployment contexts. The experiments cover both country preference rankings and utility elicitation, providing evidence from two different evaluation paradigms.
Critical weaknesses
  1. The observation that LLM preferences change under different prompts or contexts is closely related to existing research on prompt sensitivity, preference consistency, and LLM-as-a-Judge reliability. For example, [1] and [2] investigate inconsistencies in pairwise preference evaluation and propose approaches to improving evaluation reliability. Although these works study a somewhat different setting, their implications for preference consistency and ranking reliability are relevant here. The paper does not sufficiently discuss these connections or examine whether alternative preference aggregation methods would affect its conclusions. A more comprehensive discussion of related work is needed to better establish the novelty of this study.
  2. The paper considers four writing-task framings (news, Reddit, school essays, and vlogs), but these contexts are primarily introduced through prompt instructions rather than actual deployment tasks. Furthermore, changing the writing task simultaneously changes the expected audience, communication style, and potentially the criteria for making a preference judgment. Therefore, it is unclear whether the observed differences reflect changes in underlying preferences or simply task-specific response adaptation. More realistic deployment scenarios and controlled experiments isolating these factors would strengthen the conclusions.
  3. In Section 4.1, the authors restrict their statistical analysis to AB/BA-consistent decisions. However, the number of retained judgments varies substantially across deployment contexts (Table 11). For example, Claude retains 11,581 judgments in the neutral setting but only 9,077 in the Reddit setting. Excluding inconsistent judgments may introduce context-dependent selection bias and affect the resulting preference rankings. The authors should investigate whether the main findings remain consistent when all pairwise judgments are retained and positional inconsistency is explicitly modeled.
  4. Although the study contains a large number of pairwise evaluations, the experiments cover only 15 countries, six queried traits, and 50 utility outcomes. These represent relatively small subsets of the original evaluation settings. In addition, the comparisons with prior studies involve different model selections and reduced evaluation spaces. Therefore, the results do not necessarily establish that the conclusions of previous studies fail under deployment-context variation. Experiments using the original models and evaluation sets, or additional independently selected tasks and contexts, would make the claims more convincing.
  5. The paper emphasizes statistically significant ranking changes, but statistical significance does not necessarily imply substantial behavioral differences. For instance, the utility experiment reports a mean cross-context Spearman correlation of 0.96, suggesting that the overall preference structure remains relatively stable. Similarly, the reported 2.47× median utility-ratio shift may be sensitive to small fitted utility values and the use of maximum-to-minimum ratios. Although the authors provide noise-floor and fixed-anchor analyses, additional uncertainty estimates and sensitivity analyses would help establish the practical importance of these changes.

[1] Towards Acyclic Preference Evaluation of Language Models via Multiple Evaluators.

[2] TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them.

Questions for authors

See weakness part

2 comments

We thank the reviewer for their comments. Our responses to the highlighted weaknesses are as follows:

[W1]

We thank the reviewer for their paper recommendations. Our paper needs to follow the methodologies of high-profile papers proposing stable model preferences/values (Kerche et al, Mazeika et al) to duly comment on them. Thus, their counterbalanced AB/BA designs are most appropriate for our core contribution: demonstrating the impact of context dependence in established evaluations. However, we agree that deeper analysis benefits further understanding (e.g. hierarchical clustering in Appendix A.6).

Counterbalancing and filtering inconsistent country pairings insulate our data from some of the fluctuations detailed in [1]; these insulations are not examined by PGED. Regarding [2], we use LLM-as-a-judge only as an additional exploratory use-case (Appendix C), thus their method is beyond our scope. Still, we will acknowledge these other methodologies in the paper to inform future work.

[W2]

There is a clear need for context space to be better defined and mapped, we are targeting this in a follow-up study. This paper uses 5 contexts in this space that cover substantially different use-cases plus several ablation checks to address the persistent nature of context dependence (Appendices A.9-10; B.3). For country-preferences in Llama-3.3-70B-Instruct, context differences remain significant in 60/60 comparisons under answer-only elicitation and 38/60 when explicitly requesting the model’s own preference. Both show context dependence does not rely on models generating a writing task or explanation, indicating how it may manifest in realistic scenarios.

[W3]

We agree our decision-level analyses are conditional on order-consistent choices. Analyses were performed only on valid, consistent responses to eliminate ordering bias, group unevenness being a necessary byproduct already accounted for in our analyses (following the standard of human behavioural research). Nonetheless, given inconsistent responses produce net-neutral scores for each country (i.e. 0), including them does not change between-country rankings at all. The same decision-level analysis across 300 model-trait-context-pair comparisons yields significant context differences in 90.7% of comparisons, versus 73.7% with filtering. Therefore, our conclusions are clearly preserved even in the raw data.

[W4]

Though the amount of ranked items in our study is deliberately constrained compared to Kerche et al and Mazeika et al, our experiment scale is conversely increased by the repeats we ran of each pair, more than both papers and with stricter assumptions. This was critical to our research, which requires more data per preference to claim evidence of persistent context instability. While our methods could be extended to their 197 countries or 500 outcomes, random sets of just 5 of our countries or 15 of our outcomes already detect the context effect in every model, and 89% of all 197 countries already fall within the range of our 15 picked ones. Given that their findings do not generalise across our contexts, despite far fewer rank shifts being possible, we believe their conclusions remain legitimately challenged.

[W5]

We agree delineating significance and effect strength is vital for our data: alongside p-values, effect sizes are presented in Tables 5-6 of the main paper, (further detail in Appendices e.g. A.4). The Spearman’s rho noted by the reviewer refers to the broad utility domains, rather than specific value trade-offs a model may make in agentic settings. We highlight this mismatch between a model’s general ‘trends’ and its highly variable, specific choices in the paper as a risk of over-assuming stable preference mechanisms in LLMs from context-agnostic research.

We also checked our results’ sensitivity to uncertainty. Given sensitivity analysis itself can depend on post-hoc manipulation, and lies beyond the scope of this paper, we instead use bootstrapped analysis. We recalculated utilities and ratios in 1,000 paired resamples, removing pairs with a normalised mean score in either outcome of <0.1 in any context (~25%). Under our existing fitting procedure, the median ratio shift has a 95% bootstrap range of 2.34-2.67×, or 1.86-2.02× with sign-changing pairs and pairs below the 0.1 threshold excluded. The aggregate finding is thus stable to resampling, with median use preventing sizeable impact of outliers. This suggests our observed data reflects context dependence and is unlikely to be explained by random sampling alone, which can only result in a median ratio shift of 1.26×.

Finally, we examine if fitted utilities and noise scales predict observed choice frequency. Under five-fold cross-validation, the mean absolute discrepancy across five confidence bands evaluated separately per model was 1.3pp. The close match supports using this model to describe expressed preferences. We will add these results to the paper.

Thank you for the detailed response. I appreciate the additional analyses, but several concerns remain unresolved.

Following established protocols does not sufficiently address the novelty concern. The response acknowledges related work but lacks substantive comparisons with alternative preference evaluation and aggregation methods.

The ablations demonstrate context-dependent responses but do not distinguish changes in underlying preferences from task-specific adaptation. Genre instructions may still influence decision criteria, even when models explicitly report their own preferences.

The unfiltered analysis partially addresses my concern. However, the difference between filtered (73.7%) and unfiltered (90.7%) significance rates suggests that inconsistent judgments affect statistical inference. Their context-dependent impact remains insufficiently examined.

More repetitions improve statistical precision, not representativeness. Subsampling within the restricted evaluation sets and using different models cannot establish that the original findings fail to generalize.

The bootstrap analysis improves uncertainty estimation but does not establish practical significance. The reduced ratio shift (~1.9× after exclusions) and high overall rank correlation (ρ = 0.96) leave the behavioral importance of these differences unclear.

The response addresses some concerns, but the main issues remain. I will keep my original rating.

Reviewer_JYykOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines how task context affects the preferences and values expressed by large language models. The authors apply country preference ranking and utility elicitation across five contexts, including neutral comparison, news writing, Reddit posts, school essays, and vlog scripts. They keep the comparison options fixed and use repeated sampling and counterbalanced option order. The experiments cover five models, 15 countries, and 50 outcomes, with more than 1.2 million model calls.

The results show systematic changes in country rankings and estimated utilities across contexts. Across all five models, the vlog context shifts country rankings in favor of the Global South relative to neutral comparisons. Controls on selected models show that context effects persist when explanations are omitted or models are explicitly asked to express their own preferences. The authors conclude that preference and utility measurements should be interpreted in relation to their task context, with explicit testing of whether findings transfer to other contexts.

Critical strengths

The whole paper is well-structured and easy to grasp. The question is well targeted and has clear practical relevance. It identifies an insufficiently tested assumption behind preference evaluation and provides evidence that affects how such evaluations should be interpreted. I consider this a meaningful contribution to research on LLM preferences and alignment.

The experimental design is also well matched to the research question. The two main experiments support the same broad conclusion, while the supplementary checks strengthen confidence that the findings hold under different evaluation settings. I find the overall argument coherent and convincing, and the clear presentation makes the evidence easy to assess.

Critical weaknesses

I see the practical implications for model evaluation as one of the paper’s main strengths. However, the benefit of accounting for task context could be demonstrated more directly. For example, the existing experiments could compare neutral-context and target-context evaluations under the same sampling budget to assess how well each predicts held-out model choices in the target context. Alternatively, a simulated travel recommendation task could test whether context-matched preference estimates better predict the destinations a model recommends. Either experiment would help establish how the findings can improve evaluation practice. I regard this as an opportunity to strengthen the paper, and it does not change my support for acceptance.

Questions for authors

I would like to know that, within the reported experiments, how do changes in preference rankings across task contexts compare with differences between models evaluated in the same context? Can changing the task context have a larger effect than switching models? and how often does changing the task context reverse which option a model tends to choose? Distinguishing these reversals from changes in preference strength would help readers understand how frequently the reported effects lead to a different choice.

2 comments

We thank the reviewer for recognising the relevance of our work and for providing insightful suggestions in how the work could be further expanded. Our answers to reviewer’s suggestions (W1) and questions (Q1,2) are as follows:

[W1]

We agree there are practical implications connected to the behaviours we have described, as noted by the reviewer. Due to the current size of the paper and its focused methodological approach, we intend to explore this in a dedicated follow-up.

Nevertheless, to better inform this future work, we have investigated one of the proposed experiments: comparing the neutral-context and target-context evaluations under the same sampling budget to assess how well each predicts held-out model choices in the target context.

We evaluated context-matched calibration through held-out binary choice prediction using pairwise logistic classifiers. Compared with equal-budget neutral calibration, context matching reduces test cross-entropy loss by 20.8% for country preferences and 15.0% for utilities, while improving prediction accuracy by 2.34 and 1.12 percentage points, respectively.

This directly reaffirms the presumed practical implications of our findings, showing that accounting for task context improves how well preference estimates predict held-out model choices on familiar questions in the target context, even when budget is controlled for.

[Q1] How do changes in preference rankings across task contexts compare with differences between models evaluated in the same context? Can changing the task context have a larger effect than switching models?

On average, the change to preference/value rankings caused by a model change is 2.63× as large as that caused by a deployment-context change, averaging the ratios from the two experiments. This is still rather notable, as we have quite a sparse set of models with substantial differences between them. Moreover, when we look closer at specific model and context pairs, we find that in some instances a change of context has a larger effect than switching models. One such instance is that, among all 50 utility outcomes, changing Qwen3-30B-MoE’s context from school to vlog shifts the position of outcomes by an average of 4.00 positions, while switching from Qwen3-30B-MoE to Llama-3.3-70B-Instruct results in a shift of only 3.18 positions across contexts (only 2.48 in the school context). This shows that, although model changes are generally more impactful on average, the impact of context dependence is substantial enough to warrant dedicated investigation in its own right.

[Q2] …and how often does changing the task context reverse which option a model tends to choose? Distinguishing these reversals from changes in preference strength would help readers understand how frequently the reported effects lead to a different choice.

We can test this by investigating whether the same option is chosen more than 50% of the time in one context and less than 50% in another, with ties counting as non-reversals. Across the five tested contexts, we observe majority reversal in 14.4% of model-trait-country-pair settings in the country-preference experiment, (e.g. Qwen picking Brazil over Czechia for better vibes in the vlog context, but Czechia in the Reddit context). Reversals rise slightly to 17.6% of model-outcome-pair settings in the utility experiment and, for subjective country preferences, to 19.2%. This further clarifies and substantiates the real impact of deployment context.

We agree with the reviewer that these additional results are useful and will add them to the paper.

Thank you for your response. Additional analyses and explanations have adequately addressed my questions, and I also appreciate the clear narrative and well-structured presentation of the paper, which made it a pleasure to read. I understand that a more extensive investigation of the practical implications may be beyond the scope of the current paper due to the conference's page constraints. Therefore I will maintain my positive assessment and support for this work.

Best, and good luck.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.