- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper tests whether preferences elicited from language models change across neutral, news, Reddit, school, and vlog contexts. It studies country-preference rankings and pairwise preferences over outcomes used to fit utility models. Across five models and extensive repeated comparisons, context changes both local preferences and some inferred trade-offs, even when broad rankings remian comparatively stable. Additional analyses examine prompt wording, temperature, answer-only elicitation, explicit requests for the model's own preference, and the sampling-noise floor of utility-ratio variation.
Critical strengths
-
The study examines a meaningful source of variation using complementary measurements. Moving beyond surface paraphrases to intended communication contexts is useful for understanding how preference assessments travel across applications. The distinction between stable overall ordering and unstable local comparisons or fitted trade-offs provides more information than an average rank correlation alone.
-
The appendix contains controls that materially strengthen the result. Context effects persist in answer-only settings, and a country-preference control explicitly asks for preferences independent of audience or format. The utility study also compares ratio variation against a refitted permutation-based noise floor. These analyses make it less plausible that the findings are explained entirely by generated reasoning, one wording choice, or fitting noise.
Critical weaknesses
-
The decision-level estimand is conditional on order consistency. The country-preference tests retain only AB/BA-consistent decisions, and retention rates differ across contexts. Consequently, these tests concern preferences conditional on consistency instead of preferences across all valid counterbalanced responses. I recognise that complete inconsistent pairs contribute zero to summed win-loss scores, so their exclusion need not change the country rankings themselves. The remaining concern is whether the reported decision-level significance and normalized effect sizes remain representative of the unfiltered responses.
-
The mechanism and scope of the context effects remain incompletely identified. The main conclusion about context-conditioned elicited preferences is supported. However, the own-preference control in Appendix A.11 covers only one model in the country-preference task. It remains unclear whether the utility findings similarly persist when the evaluation objective is explicitly held fixed across framings. This limits interpretation of the utility shifts as changes in a model-level value system.
Questions for authors
-
Please clarify which reported statistics are affected by excluding AB/BA-inconsistent pairs. In particular, do the decision-level context effects persist in an analysis of all valid counterbalanced responses that preserves the AB/BA pairing? For summed win-loss rankings, please confirm whether filtering leaves the rankings unchanged.
-
Does an explicit fixed-objective/own-preference instruction preserve the key utility findings, or should that part of the conclusion be limited to context-conditioned communication tasks?
Official Comment by Authors (PART 1/2)
We thank the reviewer for the careful review. Below we conduct an additional analysis of our data to answer the critical weaknesses [W1,2,3] and questions [Q1,2,3]. Due to the amount of questions raised, we split our answer into two parts to properly answer them and detail how we will address them in our paper.
[W1]
We define our ratios by anchoring them to the “no change” outcome, set at 0. The absolute ratio captures changes in inter-outcome utility even when the ordering stays the same, and is unaffected by a common rescaling of the scores within each context. This does mean the ratio measures variation in relative utility under the fitted model, rather than direct preference strength. We will use the extra space in the camera-ready version to further clarify this.
We address W1’s further concerns in Q1 and Q2:
[Q1]
We report the median over the mean to reduce the influence of outlier ratios, although this does not completely negate sensitivity to near-zero scores. Removing pairs where either outcome’s average |normalised score|<0.1 in any context excludes about 25% of pairs. The median ratio shift changes from 2.47× to 1.99×, which is expected given the large number of pairs with above-average ratios removed. No zero-score outcomes were used.
We then focus on pairs whose scores change sign. In the paper, these pairs were kept to preserve information on magnitude shifts. Nevertheless, we agree that absolute ratios conceal changes in preference directions. Thus, the cleanest ratio shift value drops them. Removing sign-changing pairs shifts the median ratio shift to 2.01×, or 1.90× after applying the 0.1 threshold.
All of this shows that the reported trend is not confined to near-zero or sign-changing pairs, with the ratio shift still being significantly above the noise floor. We will add these subset results to the paper to clarify their influence.
[Q2]
To address uncertainty, we ran a bootstrapped analysis. For 1,000 resamples, we redrew with replacement the ten repeated answers per outcome pair per context, refitting utilities and recomputing ratios. Under our existing fitting procedure, the median ratio shift has a 95% bootstrap range of 2.34-2.67×, or 1.86-2.02× after excluding sign-changing pairs and pairs below the 0.1 threshold. The aggregate finding is therefore stable to resampling, with the median preventing sizeable impact of outliers. Furthermore, shuffling context labels removes their original association with the answers, reducing the median shift to 1.32× (1.26× with exclusions). This suggests that the observed shift reflects context dependence and is unlikely to be explained by random sampling alone.
We also checked whether the fitted utilities and noise scales predict observed choice frequencies. Under five-fold cross-validation, the mean absolute discrepancy across five confidence bands evaluated separately for each model was 1.3 percentage points. This close match supports using the fitted model to describe expressed preferences. We will add these results to the paper.
[W2]
We used AB/BA agreement to minimise in ordering bias elicited preferences, requiring the same country to be selected in both conditions. We agree this makes decision-level analyses conditional on order consistency. For the rankings themselves, however, complete AB/BA disagreement contributes one win and one loss to each country (i.e. a net zero). As such, country rankings and North/South gaps remain identical irrespective of inconsistent pairs being included.
For decision-level significance, retaining all complete AB/BA pairs (including disagreements) yields significant context differences in 90.7% of the 300 model-trait-context-pair comparisons. Originally this was 73.7%, using the same chi statistic and Holm-Bonferroni correction while preserving AB/BA pairing during permutation. Thus, not conditioning on agreement increases the proportion of significant context comparisons, reinforcing that context dependence does not rely on agreement filtering. As such, we retain agreement-only testing as the more conservative option.
[Q3]
Most of this question is covered in our answer to [W2]. For [Q3], we also investigate valid answers from incomplete pairs. First, we investigate their presence; extraction-failure / AB/BA-disagreement rates in the paper are:
Neutral: 0.16% / 17.83%
News: 0.49% / 21.73%
Reddit: 0.39% / 26.91%
School essay: 0.27% / 21.74%
Vlog: 0.87% / 21.21%
We examined valid responses from incomplete pairs by weighting AB and BA choice rates equally. With weighting, the results are nearly unchanged. Aggregate country ranks change by only 0.02 positions on average, and North/South gaps by 0.007 positions on average (maximum 0.10). All five models retain the subjective neutral-to-vlog shift toward the Global South.
Overall, this means that the main ranking and geographic bias conclusions remain robust across the tested data-filtering choices.