- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the behavioral value consistency of LLMs in multi-step decision-making scenarios. The authors construct a benchmark consisting of 450 branching scenarios based on 45 pairs of Schwartz's basic human values. Each scenario involves a sequence of decisions where the model must make trade-offs between competing values. The experiments cover multiple LLM families and sizes. The main finding is that models exhibit higher consistency within scenarios than across scenarios, with the gap generally decreasing for larger models. The authors further analyze whether model behavior can be explained by a one-dimensional value ranking and how different types of narrative pressure are associated with preference reversals.
Critical strengths
-
An interesting and relevant research question. Most existing evaluations of LLM value alignment focus on isolated questions or static preference rankings. This paper instead studies model decisions in multi-step scenarios, where previous choices and changing circumstances can affect subsequent decisions. I think this is a useful setting for studying LLM behavior beyond single-turn evaluations.
-
A reasonably comprehensive benchmark and experimental analysis. The benchmark covers 45 value pairs and 450 scenarios, and the authors evaluate a broad range of models. The paper also goes beyond reporting consistency scores by analyzing value-ranking structures, preference reversals, and robustness to paraphrasing. These experiments make the empirical observations more convincing.
Critical weaknesses
-
The within–across consistency gap may be partly influenced by the metric design. The two consistency metrics are calculated at different aggregation levels. Within-scenario consistency is computed from decisions within a single trajectory, while cross-scenario consistency pools decisions across multiple scenarios. Since both metrics use the majority preference proportion, shorter trajectories can naturally produce higher consistency scores, even without genuine scenario-dependent behavior. Although the authors include random baselines, I am not convinced that these fully address this issue. A more appropriate control would preserve the number of decisions in each trajectory while assuming a fixed value preference across scenarios.
-
The interpretation of cross-scenario inconsistency is not sufficiently justified. Different scenarios involving the same value pair may have very different consequences, risks, and trade-off intensities. For example, prioritizing security over benevolence in one scenario but making the opposite choice in another does not necessarily imply an inconsistent value preference. The decisions may simply reflect different circumstances. The current annotations mainly capture the direction of value preferences, rather than their relative importance or the magnitude of the trade-offs. The authors acknowledge this limitation, but I think it directly affects the main conclusion. The two human participants also exhibit a similar within–across gap, which raises the question of whether this pattern necessarily indicates a meaningful deficiency in LLM behavior. I also find the narrative-pressure analysis somewhat descriptive. Different pressure types are associated with different reversal rates, but it is unclear whether these differences are caused by the pressure itself or by other properties of the scenarios.
Questions for authors
-
Regarding the consistency metrics: Have the authors considered a baseline where each value pair has a fixed preference probability, and decisions are sampled independently using the same trajectory lengths as the benchmark? This could help estimate how much of the within–across gap arises from the metric definitions.
-
Regarding scenario comparability: How do the authors ensure that different scenarios involving the same value pair have comparable trade-off intensities? Could differences in consequences or risk levels explain part of the observed gap?
-
Regarding narrative pressure: Would it be possible to compare matched decision points with and without a particular pressure type? Such a comparison would help clarify whether the reported reversal patterns are actually related to the narrative pressure rather than other contextual factors.
3 comments
Thank you again for your thoughtful comments. We have added additional analysis and clarification. We would appreciate it if you could reconsider your assessment and score based on these responses, and would be happy to clarify any remaining questions.
Dear Reviewer Bsjb,
As the rebuttal phase is nearing its end, we would appreciate hearing whether our rebuttal has resolved the issues you raised. If any concerns remain, please let us know so that we can address them in the remaining time.
If our responses address your concerns, we would sincerely appreciate it if you could reassess your score.
Thank you for your time and thoughtful review.
Thank you for your thoughtful review and for suggesting a specific control for the metric design. We ran it as you described. We address each point below.
W1 & Q1. Yes, this is the baseline we ran (R2 asked the same). For each value pair, we now fix its preference rate and the trajectory lengths and then sample each choice (1,000 samples). The excess (observed gap - null gap) is still significant in all 19 models.
CIs are bootstrapped over value pairs; p is from the sampling null.
We will replace the raw-gap test in Table 1 with this analysis.
W2 & Q2. We do not match intensity across scenarios (there is no objective common scale for it across different stories). Actually we first scored each option's value intensity on a 1–3 scale, but annotator agreement on these scores was low as it's very subjective. So all analyses use only the direction (sign level). Instead, we make sure every option is a reasonable choice and its consequences are not exaggerated, so that no scenario collapses to one obvious option (a dedicated anti-collapse/balance item in our human evaluation: 4.90/5; Table 5, App. J).
We agree that different circumstances can lead to different choices. This is exactly our point. We do not claim that LLMs are flawed. We claim that a fixed value profile (a ranking or score), as used in prior work, cannot describe choices that depend on the scenario. Humans show the same gap, which supports this.
W2 & Q3. We agree this would be an interesting comparison, but our benchmark was not designed for it. In a branching story, the pressure is part of the plot, so changing it at one node changes the later nodes and the tree. Different subtrees follow different storylines, which makes a fair comparison difficult.