- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether preferences elicited from LLMs through pairwise choices can translate into behavioral incentives in downstream tasks. The authors first estimate model-specific utility rankings across four outcome domains and then compare generation quality when task success is associated with high- versus low-utility outcomes.
The experiments cover seven LLMs and four generation tasks, including essay writing, grant abstracts, incident postmortems, and translation. The authors find no reliable improvement in judged output quality under high-utility incentives, while explicit effort instructions, role prompting, and harmful-outcome conditions produce measurable changes in output quality.
Additional analyses examine utility magnitude, reasoning-enabled generation, and output characteristics. The authors interpret these findings as evidence of a gap between elicited preferences and their behavioral influence.
Critical strengths
The research question is important, and the empirical investigation is relatively comprehensive.
Investigating whether elicited preferences have behavioral consequences beyond pairwise choice settings is a valuable direction with potential implications for AI alignment and safety. The paper evaluates seven actor models across multiple outcome domains and generation tasks, while introducing several control conditions and robustness analyses. These experiments provide useful empirical observations about the relationship between stated preferences and downstream generation behavior.
Critical weaknesses
1. The paper's claims are not sufficiently supported by the experimental paradigm.
The paper draws broad conclusions about whether elicited preferences can function as behavioral incentives, yet primarily measures their effects on generation quality. However, a model's preference for an outcome does not necessarily imply that it will produce higher-quality text to pursue it; such preferences may instead manifest in action selection, planning, or long-horizon decision-making. Furthermore, the incentive mechanism relies entirely on hypothetical promises, without a verifiable connection between task performance and the promised outcome. The null results may therefore reflect limitations of the incentive framing rather than an absence of behavioral motivation.
Consequently, the experiments support a narrower conclusion about the lack of utility-to-quality transfer under the tested conditions, rather than the broader claims about intrinsic goals or behaviorally meaningful preferences.
2. The experimental evaluation and positive controls are insufficient to support the conclusions.
The paper relies heavily on LLM-as-a-Judge evaluations, which are known to exhibit length bias. Notably, Direct Effort Prompting increases output length, while Harmful Outcome Prompting reduces it, consistent with the directions of the reported judge preferences. Although the authors control for length in some feature analyses, they do not establish whether the original pairwise win-rate effects persist after controlling for length.
Moreover, only 3,865 of the 8,400 main comparisons are non-ties, leaving approximately 54% excluded from the win-rate calculation. This raises concerns about the judges' ability to detect subtle quality differences. Since the positive controls also introduce noticeable changes in output length and other textual features, their effectiveness does not necessarily demonstrate that the evaluation pipeline is sensitive to subtle utility-driven effects. Therefore, the positive controls do not sufficiently rule out evaluation limitations as an alternative explanation for the null results.
Questions for authors
-
Credibility of the Incentive Scenario: The incentives are presented as hypothetical promises, and Appendix G.2 shows that some models express suspicion or disbelief. For Qwen3.5 9B, only 163 of 1,920 comparisons remain after excluding such cases. How do the authors rule out the possibility that the null results are driven by models not believing the promised outcomes?
-
Length Bias in LLM-as-a-Judge: Direct Effort Prompting increases output length, while Harmful Outcome Prompting reduces it, consistent with the directions of judge preferences. Given the known length bias in LLM-as-a-Judge evaluations, do the reported effects remain significant after controlling for output length? Can the authors provide length-controlled win rates?
-
Judge Reliability and Tie Handling: Approximately 54% of the main comparisons are ties, potentially reflecting limited judge sensitivity or disagreement. Can the authors validate the results using stronger judges or human evaluations? Why are ties excluded from win-rate calculations, and do the conclusions remain robust when ties are assigned a score of 0.5?
-
Model-Specific Utilities: Appendix Table 6 shows that fitted utilities are almost perfectly monotonic in the number of lives saved within each beneficiary group. To what extent do the High–Low Utility comparisons reflect outcome magnitude rather than model-specific preferences? Can the authors test utility transfer while holding the number of beneficiaries fixed?
3 comments
Response (2/2)
Q3, W2b. Judge Reliability and Tie Handling
Evaluation validity. For validity, we primarily appeal to the textual feature analysis, which identified concrete, intuitively plausible factors associated with higher panel ratings. A fourth LLM, blind to the experimental conditions and panel verdicts, assessed task-specific dimensions such as argument depth, coherence, and evaluation rigor. Together with the generic text features, these analyses show that 88.1% of 59 feature-task combinations move in the panel-preferred direction under effort, compared with 52.5% under utility. The judges track things a motivated writer would change. The new length-control analysis above further shows that this is not simply a preference for longer outputs.
We have also manually assessed a number of the essay pairs ourselves. The differences are admittedly subtle, but not infrequently apparent to us. We have not conducted a formal human-rating study or a stronger-panel replication of these comparisons, but consider one or both a useful addition to the camera-ready copy of the paper.
Tie handling. For the main high–low, effort, and role comparisons, each of three judges evaluates both presentation orders. A judge supplies a directional verdict only if its two answers agree; a reversal or a tie in either order produces a judge-level tie. The panel then requires at least two of the three judge verdicts to favor the same condition. For example, high/high/tie is a high-side win, whereas high/low/tie and high/tie/tie are panel ties. We will clarify this in Appendix D.1.
We exclude ties in the primary analysis to measure which side wins when the panel identifies a directional difference. The following sensitivity analysis retains every resolved comparison, scoring wins as 1, ties as 0.5, and losses as 0, with the same equal-cell aggregation and bootstrap.
| Comparison | Panel ties | Win rate, ties excluded [95% CI] | Win rate, ties as half-wins [95% CI] |
|---|---|---|---|
| High vs low utility | 54.0% | 51.2% [48.7–53.6] | 50.6% [49.5–51.7] |
| Strong vs normal effort prompt | 49.1% | 76.8% [71.8–81.6] | 64.2% [61.2–67.2] |
| World-class vs skilled role | 55.1% | 61.2% [57.1–65.6] | 54.8% [53.1–56.7] |
| Harmful vs no-outcome | 7.0% | 40.5% [35.7–45.1] | 40.9% [36.3–45.4] |
Thus, our conclusions are robust to how ties are treated. The lower tie rate for harmful versus no-outcome reflects a procedural difference: that comparison used one randomized presentation order per judge, whereas the other comparisons required agreement across both orders. (This is a historical artifact, not a deliberate choice.)
Regarding sensitivity, the high–low interval excludes an average advantage exceeding 3.6 percentage points above chance. With ties counted as half-wins, the upper bound is 1.7 points above chance.
Q4. Model-Specific Utilities
Count and beneficiary identity. To address this question, we decomposed the fitted utilities into count, identity, and their interaction. Count accounts for the large majority of fitted utility variance in religion and countries (99.5% and 98.7%, respectively), but only 54.9% for animals.
We then tested whether the fitted utilities predict held-out choices with identical counts and different beneficiaries:
| Domain | Same-count choice-prediction accuracy [95% CI] |
|---|---|
| Religion | 59.5% [50.7–69.2] |
| Animals | 91.2% [89.3–93.1] |
| Countries | 80.8% [77.7–83.8] |
So there is predictive information about beneficiary identity in the utilities in all three domains.
Does that identity preference transfer to output quality? We have now run the requested fixed-count comparison for religion and countries across all seven models, selecting each model's highest- and lowest-ranked beneficiaries at the largest fitted count: 1,000 lives for religion and 60 for countries. We used the original 30 essay topics with seven repeats, retaining the same generation settings and both-orders judge panel. Only beneficiary identity changes within each pair. Combined with the existing same-count animal comparisons, the results are:
| Domain | Lives saved in each arm | Tasks | Non-tied pairs | Higher-utility-side win rate [95% CI] |
|---|---|---|---|---|
| Religion | 1,000 | Essay | 577 | 52.9% [48.2–57.7] |
| Countries | 60 | Essay | 550 | 52.2% [44.5–59.5] |
| Animals | 1, 2, or 60; matched within each pair | All four | 130 | 47.2% [33.9–60.4] |
(These use equal-cell averages and the paper's two-stage bootstrap, with ties excluded.)
Thus, the paper's utility null remains when count differences are excluded.
We thank the authors for their detailed response. In particular, the ablation study on tie handling and the length-controlled experiments have effectively addressed our concerns regarding the experimental setup. However, our concern regarding Weakness 1 has not been adequately resolved. Therefore, we have decided to maintain our original score (3: Weak acceptance).
Response (1/2)
We thank the reviewer for their careful consideration and appreciate the opportunity to engage and improve our paper. We also apologize for the late reply; various exigencies made it impossible to address this last week.
W1. Supported Claims
We agree that elicited preferences may manifest in other ways; as we say in the paper’s final paragraph, “It is possible that untrained preference seeking behavior could manifest in environments we have not tested or towards preferences we have not measured.” Nonetheless, we do propose that our findings establish that these preferences do not function as incentives in four tasks where (1) they intuitively should if they have motivational force and (2) exogenous cues (prompts) empirically do motivate better outputs. Other paradigms in the literature entail explicit choices about outcomes; the behaviors measured are overt affirmations of the preferences. In contrast, our paradigm tests the incentive value of the preferences while the measured behavior itself is independent (and held constant across outcomes). To our knowledge, these are the first tests of LLM preferences as behavioral incentives on tasks where models are able to modulate their behavior in response to other cues.
We offer analysis below to further support our argument that doubt about the promised outcomes is not driving the null results; however we acknowledge some uncertainty on that point and will make that more salient in the paper's text.
Q1. Scenario Belief and Suspicion
Regarding Qwen3.5 9B's high expressed disbelief/suspicion rates (19.2% and 76.5%, respectively; 78.0% combined), note that the expressed disbelief/suspicion rates for the other six models were rather modest, and that, for all seven models, excluding those pairs had essentially no impact on the high-side utility win rate. In fact, excluding all pairs except those where the high-utility side explicitly expressed belief also had essentially no impact on win rates. Thus we suggest that disbelief in the outcome was not common in most models, and that, regardless, overt belief status did not influence performance (because, we infer, the "preferred" outcome does not have intrinsic motivational value to the models).
Q2, W2a. Length-Controlled Win Rates
To address the impact of length, we tested the panel win-rate results in two ways:
The table below summarizes the results:
(Entries are win rates for the first-listed side [95% bootstrap CI]. Both analyses retain the paper's equal-cell weighting and exclude panel ties. Confidence intervals use 10,000 bootstrap resamples of cells and then pairs within cells; the regression is refitted in each resample. Every cell remains represented in the similar-length subset.)
Under both controls, all of the paper's main effects remain substantially unchanged.
We thank the reviewer for raising this question. We believe this strengthens the paper's results and will add it to the text.