- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ValueGraph, a graph-based approach for studying how language models make trade-offs between competing values. By building 2,791 scenarios covering Schwartz’s 19 refined values and eight decision domains, the authors represent models’ value orientations and pairwise priorities in a unified graph structure. Experiments on 22 language models show that models with similar overall value rankings may still differ substantially in specific trade-offs, and that some value priorities change depending on the decision context. Through robustness checks and a prompting study, the paper further demonstrates that ValueGraph can help analyze value alignment beyond simple value rankings.
Critical strengths
1. The paper identifies a meaningful limitation of global value rankings. In Section 5.1, Qwen 3.8 Max and Claude Opus 5 achieve a highly similar overall ranking (ρ = .960), yet show differences of roughly 35–37 percentage points on specific trade-offs involving Self-direction–Thought versus Face and Achievement. This demonstrates that similar global value profiles can still correspond to different decision behaviors, motivating the need for evaluations that preserve local value conflicts rather than collapsing them into a single ordering.
2. The dataset design enables systematic analysis of both shared patterns and contextual variation. By covering all 171 value pairs and including different third-value combinations and domains, the dataset supports robustness analysis beyond isolated conflicts. The shared six-by-five value boundary identified in Section 5.2 provides a more precise characterization of common model tendencies, while the domain-level analyses reveal where priorities change across contexts.
Critical weaknesses
1. The incremental contribution over related work is not sufficiently clear. Prior work, including ConflictScope (Liu et al., 2025) and Priority Graph (Tang et al., 2026), has explored value conflicts, contextual priorities, and graph-based representations. ValueGraph extends these directions with broader value-pair coverage and robustness analysis. However, the paper does not clearly establish what new insights are enabled by the graph representation itself beyond organizing pairwise preferences and contextual information. The findings on hidden local trade-offs and domain-dependent priorities are valuable, but the contribution would be stronger if the paper demonstrated cases where ValueGraph enables analyses or interventions that simpler pairwise evaluations would miss.
2. The validation does not fully establish whether the reported trade-offs are robust across elicitation protocols. Section 4.2 mainly validates BWS through correlations of overall value rankings. However, the main claims concern pairwise priorities and domain-specific trade-offs, which may not be captured by ranking-level agreement alone. Since BWS measures consideration while protocols such as P3 measure direct choices, evaluating pairwise relations across protocols would provide stronger evidence that the reported trade-offs reflect stable model behavior rather than properties of the BWS setup.
Questions for authors
1. What does ValueGraph add beyond ConflictScope and Priority Graph? In particular, which of the paper’s findings rely on the graph representation or robustness analysis, rather than on the underlying pairwise preferences and domain-level analyses?
2. For the models evaluated with both BWS and P3, how consistent are the resulting pairwise priorities across protocols? Do the shared value boundaries and domain reversals observed with BWS also hold under P3, or are they sensitive to the elicitation method?
3. In Section 6, which parts of the context-rule prompt are derived from ValueGraph analysis, and which are specified by the target policy? Does the experiment demonstrate that ValueGraph helps identify alignment targets, or mainly that models can follow an explicitly provided rule?
7 comments
Responses to Reviewer L4J9 (2/4)
1. (W1, Q1) Contribution over Related Work (continued)
W1-2. "... the contribution would be stronger if the paper demonstrated cases where ValueGraph enables analyses or interventions that simpler pairwise evaluations would miss."
An evaluation that reports one win rate per pair cannot tell which differences, shared priorities and reversals survive a change of materials. The domain-conditioned edges also support an intervention: they locate the two domains where the alignment target of §6 has to differ. There, in both models of §6, a context-free prompt raises agreement with the target when the two domains are pooled (by 16.2 and 9.8 points), yet lowers it in Health (by 22.2 and 7.5 points; see Q3).
(c) What new findings are enabled by these components?
Our three principal findings illustrate their distinct contributions.
Hidden local differences (§5.1). Qwen 3.8 Max and Claude Opus 5 exhibit highly similar global value rankings (Spearman ρ = .960), yet differ by 37.3 and 35.3 percentage points on specific trade-offs involving Self-direction–Thought. These differences become visible when empirical pairwise priorities are retained rather than summarized solely through global rankings.
Robust shared structure without a universal total ordering (§5.2). Across 22 models, we identify 61 relations with consistently retained directions, including a complete 30-edge bipartite subgraph in which six favored values precede five others. None of the 15 relations within the favored group is robustly retained across all models. This reveals a shared priority boundary alongside unresolved internal trade-offs: information that a single global value ranking cannot adequately characterize.
Persistent domain-associated reversals (§5.3). By examining conditional edge priorities and their supporting materials, we identify 242 persistent reversals, twice the 121 found when domain labels are permuted. This allows us to locate which specific value relations change direction and in which decision domains, rather than reporting only aggregate contextual variation.
Q1-2. "... which of the paper's findings rely on the graph representation or robustness analysis, rather than on the underlying pairwise preferences and domain-level analyses?"
All three findings rely on the robustness analysis, and §5.1 and §5.2 also on the graph representation. §5.1 compares two graphs at both levels. Their node rankings agree, while the robustness margin shows that Claude Opus 5 retains Self-direction–Thought over Face and over Achievement and Qwen 3.8 Max retains neither direction. An edge is retained when its favored value wins at least 60% of comparisons in each of the 26 material checks. The 985 between-model gaps that §5.1 reports as significant after family-wise correction all keep their sign under each of these checks. In §5.2, the boundary consists of edges retained in every one of the 22 models, and its form is a property of the graph: a complete six-by-five bipartite subgraph whose favored side has no edge retained in every model. In §5.3, a reversal is counted only if it persists when any single scenario is deleted.
In summary, ValueGraph's contribution is the joint estimation of global orientations, empirical priority relations, and edge-specific robustness, enabling structural and context-sensitive analyses that neither global ranking summaries nor conceptual priority graphs provide. We will make these distinctions explicit in the main text and add a direct methodological comparison with ConflictScope and Priority Graph.
Responses to Reviewer L4J9 (3/4)
2. (W2, Q2) Consistency of Pairwise Priorities across Protocols
W2. "Section 4.2 mainly validates BWS through correlations of overall value rankings. ... evaluating pairwise relations across protocols would provide stronger evidence that the reported trade-offs reflect stable model behavior rather than properties of the BWS setup."
Q2. "... how consistent are the resulting pairwise priorities across protocols? Do the shared value boundaries and domain reversals observed with BWS also hold under P3, ..."
Thank you for the suggested analysis. The reported trade-offs are not properties of the BWS setup. As a control, we presented the same ValueGraph scenarios as direct choices between two actions (P3 in §4.2) and rebuilt the graph from these choices alone: the retained priorities, the shared boundary and the domain effects all reappear (Table 2). Both graphs use the same scenarios, the paper's estimator and its robustness margin, on the four models of §4.2. They differ only in what is asked: BWS asks which of three actions deserves most and least consideration, the P3 control which of two actions to choose. An edge is retained when its favored value wins at least 60% of comparisons in each of the paper's 26 material checks.
Table 2. Graphs built from BWS and from the P3 control on the same scenarios.
| Claude Sonnet 5 | DeepSeek V4 Pro | Gemini 3.7 Flash | GPT-5.6 Sol | |
|---|---|---|---|---|
| Correlation of edge strengths, BWS vs. P3 control (r, 171 pairs) | .975 | .894 | .980 | .968 |
| Edges retained under BWS that keep their direction under the P3 control | 126/126 | 105/107 | 121/121 | 115/116 |
| Shared-boundary edges (§5.2) with the same direction under the P3 control | 30/30 | 30/30 | 30/30 | 30/30 |
Of the 470 edges retained under BWS, 467 keep their direction under the P3 control and none is retained in the opposite direction, and all 30 boundary edges keep their direction in all four models. In these four models, holding the triple fixed, the priority differs across domains (F test, p < .05) for 27 of the 171 pairs under BWS and 35 under the P3 control, against 8.6 expected by chance. The two sets share 17 pairs, against 5.5 if they were unrelated. The domain shifts agree between the two formats (r = .82 over pair-by-domain cells). A cell's shift is the pair's win rate in that domain minus the average of the same triples across domains. The Section 6 contrast is also recovered. Leaving out one triple at a time, Health and Personal Development are selected as the domains where Conservation gains and loses most in all 600 fits under the P3 control. They differ by 22.0 points on the left-out triples.
We also ran the reverse control, keeping the BWS question and replacing the scenarios. Three models other than the paper's generator (DeepSeek V4 Pro, Gemini 3.1 Pro and GPT-5.6 Sol) wrote 1,791 new scenarios for the same value triples and domains, and eight of the evaluated models answered them, the four of Table 2 among them (Table 3). Of the 962 edges the paper retains for these eight models, 926 keep their direction, 20 reverse and 16 are tied, and all 30 boundary edges keep theirs in each of the eight models.
Table 3. The same edges on the 1,791 new scenarios, for the four models of Table 2.
| Claude Sonnet 5 | DeepSeek V4 Pro | Gemini 3.7 Flash | GPT-5.6 Sol | |
|---|---|---|---|---|
| Edges retained in the paper that keep their direction on the new scenarios | 124/126 | 101/107 | 112/121 | 112/116 |
| Shared-boundary edges (§5.2) with the same direction on the new scenarios | 30/30 | 30/30 | 30/30 | 30/30 |
BWS remains the main protocol for two reasons. It follows the best–worst scaling design of human value research [5, 6], and one BWS answer yields the three pairwise decisions that the P3 control collects with three separate questions, so each scenario needs one question instead of three. We will add this comparison of BWS and the P3 control to the appendix of the revised version.
Responses to Reviewer L4J9 (1/4)
Thank you for highlighting the strengths of our finding on what global value rankings hide and of our systematic dataset design. Below, we respond to each weakness and question.
1. (W1, Q1) Contribution over Related Work
The key distinction is that ConflictScope primarily elicits and aggregates value preferences, Priority Graph provides a conceptual representation of priorities, whereas ValueGraph develops a quantitative, robustness-aware framework for analyzing the structure of individual value trade-offs. The differences are reflected in both the methodology and the resulting findings.
Table 1. How each work treats the priority between two values.
(a) ConflictScope: Binary Conflicts and Aggregate Rankings vs. Systematically Characterized Pairwise Relations.
ConflictScope [1] generates two-value conflicts and primarily investigates how model preferences differ between multiple-choice and simulated-user open-ended evaluations, summarizing these preferences through Bradley–Terry rankings [3]. Its domain and prompt-template analyses assess the consistency of preferences across evaluation conditions.
ValueGraph advances this approach in three ways:
These differences change the analytical output: rather than only characterizing a model's overall value ranking or its stability across evaluation settings, ValueGraph identifies the precise relations that are stable, contested, or context-sensitive.
(b) Priority Graph: Conceptual Formalization vs. Measurable and Testable Priority Structures.
Priority Graph [2] formalizes context-dependent priority relations among instructions and values, but leaves the measurement function general and primarily develops conceptual arguments concerning dynamic priorities and priority hacking.
ValueGraph makes these relations empirically measurable. We define explicit estimators for pairwise priority strengths, material robustness margins, and graph-level distances; link each relation to its underlying scenarios and responses; and instantiate the framework across 2,791 scenarios and 22 models.
This transforms a conceptual priority representation into a quantitative auditing framework capable of testing cross-model agreement, identifying robust relational structures, and locating context-dependent changes.