acceptodds
Under review as a conference paper at ICLR 2027

Constitutional Value Potentials: Reading Priority Margins and Steering Value Trade-offs in Language-Model Activations

Abstract

Model constitutions specify how a language model should trade off values such as helpfulness, harmlessness, and honesty when they conflict. We ask whether different value conflicts share internal structure. We introduce Constitutional Value Potentials (CVP), which assigns each value a loading on a shared activation basis and reads a conflict through the difference of two loadings. Jointly learned loadings compose readouts for pairs without their labels. Across five instruction-tuned models from three families, rank-one composed readouts achieve mean within-orientation AUROC of -. They approach readouts fitted to each target pair on conflicts with helpfulness or autonomy and are weaker on protective-value pairs. With target-pair supervision, the shared axis also supports early monitoring: prompt-state pooled AUROC is – versus – for a text classifier, with transfer to requests without an explicit priority clause. Steering along composed directions shifts judged trade-offs in the intended direction on all eight evaluated pairs in Qwen2.5-7B.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, Representation learning, Activation Steering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RRS1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary
  • The authors aim to understand the internal structure of conflicts between different values that a language model might face during deployment, as measured by model activations. This could enable evaluating model adherence to new value pairs without having to collect additional data.
  • To do so, they fit value-specific loadings on a basis learned from model activations on different source pairs, and find that the difference in loadings between two values is reflective of the conflict. Critically, this is not directly computed using the conflicting pair, which the authors use to claim generalization from composing individual value vectors.
  • The authors also show that these directions can be used to steer models toward each individual value within a conflicting pair (for Qwen2.5-7B across eight pairs), and that they form a low-rank subspace in the model activations. However, they find limited generalization across different types of value conflicts, such as those involving protective values.
Critical strengths

While past work has extracted behavior-specific activations, the observation that behavioral activations can be composed and used to steer model behavior in new contexts is understudied in past work. A method that does this would be useful for constitution design if sufficiently validated across values.

Critical weaknesses
  • Evaluation relies on unvalidated manual prompts despite the existence of a number of existing established value conflict benchmarks, including “DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Life” (Chiu et al, ICLR 2025), “Will AI Tell Lies to Save Sick Children? Litmus-Testing AI Values Prioritization with AIRiskDilemmas” (Chiu et al, ICLR 2026), “Generative Value Conflicts Reveal LLM Priorities” (Liu et al, ICLR 2026), “CLASH: Evaluating Language Models on Judging High-Stakes Dilemmas from Multiple Perspectives” (Lee et al, ICLR 2026), “Stress-testing model specs reveals character differences among language models” (Zhang et al, 2025), etc., which are not referenced. It’s unclear how the final list of values was decided, how the prompts were created, and how the authors validated that the prompts appropriately tested adherence to the values, which are all central to the evaluation. Using existing benchmarks could allow the authors to scale the method beyond the relatively small number of pairs evaluated.
  • It’s not clear that the proposed method is identifying any fine-grained differences between values beyond a high-level helpfulness-harmlessness direction. The results seem very low-rank (the authors show that rank-2 does not improve upon rank-1), and helpful and protective values seem to generally map onto the sign on a single axis. While the authors show that the effect is not just refusal, there are many other ways to softly comply or not comply with a request. Results showing generalization to other latent bases of LLM values would be more interesting and novel.
  • The paper is very technically dense and difficult to parse.
Questions for authors
  • Why was a bag-of-words baseline used in Section 5.2, rather than stronger/more modern baselines such as sentence embedding models or existing value-specific extraction methods such as persona vectors?
  • How does the proposed method compare to just using low-rank representations of raw activations (e.g. via dimensionality reduction on the set of pooled activations)?
  • How was the LLM judge (Kimi K2.6) validated?
3 comments
AuthorsOpenReview ↗

Author Response to Reviewer RRS1 (Part 1/2)

Thank you for the thoughtful review. We appreciate your recognition of compositional transfer as the potentially useful contribution, and the concrete suggestions about evaluation and baselines. We agree that the presentation should make the scientific question and supervision boundaries easier to follow. Below we respond point by point to each concern.

R1: The Method in Three Steps\color{#c9782b}{\textbf{R1:\ The\ Method\ in\ Three\ Steps}}

W3:\color{#c9782b}{\textbf{W3:}} The paper is very technically dense and difficult to parse.

Response:\color{#c9782b}{\textbf{Response:}} The method can be summarized in three steps: fit directions for observed conflicts, extract their shared basis, and fit value loadings whose differences reconstruct those directions. A held-out pair is then read through the difference of its two loadings. The central experiment asks whether this construction predicts outcomes when that pair's labels were unavailable for fitting and calibration.


R2: Relation to Existing Value-Conflict Benchmarks\color{#c9782b}{\textbf{R2:\ Relation\ to\ Existing\ Value{-}Conflict\ Benchmarks}}

W1:\color{#c9782b}{\textbf{W1:}} Evaluation relies on unvalidated manual prompts despite the existence of a number of existing established value conflict benchmarks [...] which are not referenced. It's unclear how the final list of values was decided, how the prompts were created, and how the authors validated that the prompts appropriately tested adherence to the values, which are all central to the evaluation. Using existing benchmarks could allow the authors to scale the method beyond the relatively small number of pairs evaluated.

Response:\color{#c9782b}{\textbf{Response:}} The cited datasets are relevant, and their connection to our work deserves explicit treatment. In particular, ConflictScope$^{[1]}$ evaluates priorities through generated value-conflict scenarios, and CLASH$^{[2]}$ studies value reasoning and steerability. Our contribution concerns the activation-space composition of relations across held-out value pairs. Existing scenarios could support this question with an appropriate value mapping, source graph, and experimental split; their relevance is not in dispute.

Our dataset provides a controlled implementation of that test. It uses ten operationally defined assistant-relevant values, with the sixteen-pair graph including six additional protective-value pairs that challenge the dominant shared component. Requests are concrete tasks, and the user request remains unchanged when the priority instruction is reversed. The expanded construction pipeline includes model-assisted drafting, screening for task form and intended conflict, and manual additions. These are construction controls, not independent human validation. The complete prompt and rubric materials are already assembled; the missing access link and insufficient explanation of provenance are presentation and reproducibility shortcomings.

References: [1] Liu et al., "Generative Value Conflicts Reveal LLM Priorities," arXiv:2509.25369. [2] Lee et al., "CLASH: Evaluating Language Models on Judging High-Stakes Dilemmas from Multiple Perspectives," ICLR 2026.


R3: What the Low Rank Establishes\color{#c9782b}{\textbf{R3:\ What\ the\ Low\ Rank\ Establishes}}

W2:\color{#c9782b}{\textbf{W2:}} It's not clear that the proposed method is identifying any fine-grained differences between values beyond a high-level helpfulness-harmlessness direction. The results seem very low-rank (the authors show that rank-2 does not improve upon rank-1), and helpful and protective values seem to generally map onto the sign on a single axis. While the authors show that the effect is not just refusal, there are many other ways to softly comply or not comply with a request. Results showing generalization to other latent bases of LLM values would be more interesting and novel.

Response:\color{#c9782b}{\textbf{Response:}} We agree that a dominant fulfillment-versus-protection tendency is a plausible interpretation. Low rank does not establish a separate semantic direction for every value. However, it also does not imply that all protective-value distinctions are unrepresented. Source-only honesty–privacy transfers, while fairness–privacy exposes substantial failures:

PairModelSource-only AUROC
honesty–privacyQwen2.5-7B0.86
honesty–privacyQwen2.5-14B0.93

The rank-two ablation covers five eligible pairs from the ten-pair graph; its small average gain does not establish that every value conflict is intrinsically one-dimensional. Our evidence supports partial composition and identifies its limitations.

AuthorsOpenReview ↗

Author Response to Reviewer RRS1 (Part 2/2)

R4: Baselines and Alternative Representations\color{#c9782b}{\textbf{R4:\ Baselines\ and\ Alternative\ Representations}}

Q1:\color{#c9782b}{\textbf{Q1:}} Why was a bag-of-words baseline used in Section 5.2, rather than stronger/more modern baselines such as sentence embedding models or existing value-specific extraction methods such as persona vectors?

Q2:\color{#c9782b}{\textbf{Q2:}} How does the proposed method compare to just using low-rank representations of raw activations (e.g. via dimensionality reduction on the set of pooled activations)?

Response:\color{#c9782b}{\textbf{Response:}} Bag of words is an interpretable lexical control that receives the same request, clause orientation, and response prefix. It tests whether a simple surface-text predictor explains the monitoring result. It is not evidence of superiority over modern sentence encoders. Likewise, the main composition claim does not depend on beating this baseline: it is evaluated through held-out-pair prediction, target-pair-fitted references, and activation interventions.

Persona vectors are relevant precedents for extracting behavioral directions. CVP additionally imposes the relation wij≈A⊤(λi−λj)w_{ij} \approx A^{\top}(\lambda_i - \lambda_j) across pairs and tests the resulting held-out prediction. This is a distinct structural question, not a demonstrated head-to-head advantage over persona vectors.

Raw-activation PCA is also a different objective: it preserves activation variance, whereas our SVD operates on normalized, outcome-fitted source directions. The latter emphasizes variation associated with conflict outcomes. This methodological distinction motivates the construction; it does not substitute for an empirical claim that CVP outperforms PCA.


R5: Independent Judge and Human Validation\color{#c9782b}{\textbf{R5:\ Independent\ Judge\ and\ Human\ Validation}}

Q3:\color{#c9782b}{\textbf{Q3:}} How was the LLM judge (Kimi K2.6) validated?

Response:\color{#c9782b}{\textbf{Response:}} The generator remains frozen. Kimi applies explicit definitions and a behavioral rubric without receiving the system priority. The submission reports that alternative-rubric re-judging preserves 98.6% of decisive binary labels, which establishes within-judge stability. To validate the judge beyond this check, we added an independent-judge and human validation on Qwen2.5-7B in the instruction context of the sixteen-pair graph. Claude Haiku 5.5 re-judged all 12,480 responses with the same prompt and rubric. During the rebuttal period, two annotators from our group independently labeled a pool of 1,000 responses, blind to the system priority and to the judge scores. From this pool we sampled 300 responses from three helpfulness/autonomy pairs and three protective-value pairs (25 per pair and clause orientation). Each label marks the response as preserving the first value more, the second value more, or neither clearly. We report label agreement, and the AUROC of readouts fitted and calibrated on Kimi labels when evaluated against each judge's labels.

ComparisonResponsesDecisive-label agreementCohen's κ (three-way)
Kimi vs. Claude Haiku 5.512,4800.870.74
Kimi vs. human3000.780.61
Claude Haiku 5.5 vs. human3000.810.65
Human vs. human3000.840.70

Comparisons with human labels average over the two annotators.

Readout (fitted on Kimi labels)PairsKimi labelsClaude Haiku 5.5 labelsKimi labels (annotated subset)Human labels (annotated subset)
Source-only composition, full responseHelpfulness / autonomy0.8210.7930.780.762 [0.691, 0.827]
Source-only composition, full responseProtective-value0.7080.6810.680.637 [0.558, 0.713]
Target-supervised monitoring, prompt stateThree monitoring pairs0.8470.8120.790.771 [0.703, 0.832]

The first two label columns cover all eligible Qwen2.5-7B pairs. The annotated-subset columns cover the six annotated pairs and use responses on which both annotators give the same decisive label; human-label AUROC reports request-bootstrap 95% intervals.

Haiku agrees with Kimi on 87% of decisive labels (κ = 0.74), and Kimi's agreement with the annotators (κ = 0.61) is close to inter-annotator agreement (κ = 0.70). Readouts fitted on Kimi labels lose 0.027–0.035 AUROC under Haiku labels. On the annotated subset, human-label AUROC is 0.018–0.043 below Kimi-label AUROC and remains above chance in all three settings. The largest gap and the lowest value are on protective-value pairs (0.637 [0.558, 0.713]), consistent with the weaker composition reported for these pairs.

The supported contribution is a lightweight method that composes useful readouts without target-pair supervision, together with evidence identifying where its shared representation succeeds and breaks down.

Thanks to the authors for their response. Unfortunately, the rebuttal does not meaningfully address my major reservations with the submission (which I elaborate below), so I am maintaining my recommendation of clear reject.

R1 (Clarity): To be clear, I understood the method but feel that the paper is incredibly difficult to parse. This risks seriously limiting the paper's usefulness to the broader ICLR community. For example, the paper frequently invents new technical terms with insufficient definition that it only uses sparingly (e.g. "source-selected full-response layers"). Additionally, quantitative results are frequently presented in quick succession without substantial analysis, making it difficult for the reader to draw clear conclusions from the work (e.g. "Extrapolation from the ten-pair graph predicted non-manipulation above wellbeing, confirmed in Qwen2.5-7B and 14B, but placed honesty and fairness above privacy in the opposite order to the observed directions. Privacy and fairness each had only one source edge. For honesty–privacy, the ten-pair graph gives the wrong sign in 88% of request-bootstrap replicates at Qwen2.5-14B and 85% at Mistral. The sixteen-pair graph improves recovery (Figure 6b). These extrapolations use the ten-pair graph to predict new relations. Section 5.1 instead holds out one pair at a time from the sixteen-pair graph.") The rebuttal re-summarizes the paper method, which does not address this concern.

R2 (Relation to Existing Benchmarks): It's still not clear to me why the authors created their own benchmark rather than using existing established value conflict benchmarks, as the authors explicitly concede that "Existing scenarios could support this question with an appropriate value mapping, source graph, and experimental split" in the rebuttal. Additionally, critical details about the dataset construction process remain unknown. The authors acknowledge in their response that this is a "presentation and reproducibility shortcoming", but have not revised the paper or provided methodological details that would clarify this in the rebuttal. While they do promise to include more information about this in a future revision, we need to be able to understand how the benchmark was actually constructed in order to assess whether a paper built on the benchmark should be accepted, so promising to clarify this later does not meaningfully address my concerns.

R3 (Low Rank): I originally wrote that "It's not clear that the proposed method is identifying any fine-grained differences between values beyond a high-level helpfulness-harmlessness direction." This was based on the paper's Figure 4(a), which suggested that a rank-1 version of the method performed similarly to a rank-2 version of the method, and even outperformed it in some cases. Showing a single example of the method predicting a different conflict does not meaningfully refute this, and the authors concede that "a dominant fulfillment-versus-protection tendency is a plausible interpretation", so I still have this concern.

R4 (Baselines): To assess whether Constitutional Value Potentials is a useful method for studying the internal structure of different value conflicts, the authors should compare their method to strong existing baselines that could be applied to the same task. In my review, I suggested that the existing baseline (bag-of-words) was insufficient, and that directly comparing to persona vectors, low-rank representations of raw activations, and sentence embedding models without privileged access to activations would strengthen this result. The authors have replied by describing the baselines I suggested, but have not actually evaluated them on the paper's tasks, so this doesn't address my concern at all.

R5 (Judge Validation): The comparison to human-annotated responses is genuinely useful, and I appreciate the authors clarifying that their LLM judge was validated. I'd recommend that the authors revise the paper to include this information.

Based on these major unresolved concerns, I continue to believe that this paper should not be accepted to ICLR.

Reviewer_3uu8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Constitutional Value Potentials (CVP), which represents pairwise value conflicts through differences between value loadings on a shared activation basis. It evaluates whether readouts and steering directions can be composed without target-pair labels, and separately studies target-supervised monitoring and context transfer. Experiments across five language models show partial compositional transfer, while steering experiments on Qwen2.5-7B demonstrate shifts in judged value trade-offs. The paper also analyzes failures associated with sparse source relations and weak alignment with the shared axis.

Critical strengths
  1. Simple and lightweight formulation. CVP uses a shared low-dimensional basis and value-loading differences to construct pairwise readouts without updating the underlying language model.

  2.  Multiple useful applications. The paper demonstrates held-out-pair prediction, early monitoring with context transfer, and activation steering, illustrating several uses of the shared representation.

Critical weaknesses
  1. Evaluation relies on 379 author-written requests with limited scenario diversity. The manuscript provides no link to the complete dataset, limiting reproducibility and assessment of generalization to realistic value conflicts. The added benefit of value-loading differences over a simpler shared-axis approach remains unclear, particularly in the rank-one setting.

  2. Evaluation relies on a single LLM judge without human or independent-model validation. Excluding near-tie responses also limits conclusions about ambiguous conflicts. Steering is tested on one model at one layer, with some overlap between evaluation and intervention-selection requests and no uncertainty intervals in the main steering results.

Questions for authors
  1. Can the authors provide more details on dataset construction and evaluate CVP on an existing public dataset(such as World Values Surve) involving different values?

  2. What additional benefit do value-loading differences provide over a simpler shared-axis baseline, particularly at rank one?

  3. Do the findings hold with independent judges and steering evaluations across additional models and layers, using held-out requests and uncertainty estimates?

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer 3uu8 (Part 1/2)

Thank you for the thoughtful review. We appreciate your recognition of the simple formulation and the multiple applications of the shared representation. Below we respond point by point to each concern and clarify the distinctions you raised.

R1: Loading Differences Enable Composition Without Target Labels\color{#c9782b}{\textbf{R1:\ Loading\ Differences\ Enable\ Composition\ Without\ Target\ Labels}}

W1:\color{#c9782b}{\textbf{W1:}} The added benefit of value-loading differences over a simpler shared-axis approach remains unclear, particularly in the rank-one setting.

Q2:\color{#c9782b}{\textbf{Q2:}} What additional benefit do value-loading differences provide over a simpler shared-axis baseline, particularly at rank one?

Response:\color{#c9782b}{\textbf{Response:}} The benefit of value loadings at rank one is composition under restricted supervision. Let z(x)z(x) denote the shared activation coordinate. CVP predicts a pair through sij(x)=(λi−λj) z(x)s_{ij}(x) = (\lambda_i - \lambda_j)\, z(x). We agree that, with a fixed axis and freely fitted target-specific calibration, a nonzero loading difference can be absorbed into the calibration coefficient. Those supervised results therefore demonstrate the usefulness of the shared axis, not an additional expressive advantage of loading differences.

The main source-only experiment addresses a different question: target-pair outcome labels are excluded from basis and loading estimation, layer selection, and calibration. Relations learned from other pairs determine how the held-out pair reads the shared coordinate. The existing steering comparison makes this supervision distinction concrete:

DirectionUses target-pair labelsMean signed swing
Shared axisYes44.1
ComposedNo42.8\color{#c9782b}{42.8}
RandomNo3.2

These are point estimates, not a claim of statistical equivalence. They illustrate the intended benefit: constructing useful interventions without fitting to the target pair's outcomes.


R2: Dataset Construction and Reproducibility\color{#c9782b}{\textbf{R2:\ Dataset\ Construction\ and\ Reproducibility}}

W1:\color{#c9782b}{\textbf{W1:}} Evaluation relies on 379 author-written requests with limited scenario diversity. The manuscript provides no link to the complete dataset, limiting reproducibility and assessment of generalization to realistic value conflicts.

Q1:\color{#c9782b}{\textbf{Q1:}} Can the authors provide more details on dataset construction and evaluate CVP on an existing public dataset (such as World Values Survey) involving different values?

Response:\color{#c9782b}{\textbf{Response:}} The dataset supports controlled identification of this transfer. It contains 379 base requests across 16 pairs and ten operationally defined values. The same request is evaluated under opposite priority instructions, and request-level splitting keeps its repeated responses together. The expanded construction process includes model-assisted drafting and screening alongside manually written requests; "author-curated" is the more precise description. The complete prompt definitions, policy variants, assembly code, and rubrics are already packaged in the reproducibility materials. The missing accessible link is a reproducibility omission.

Public datasets can provide valuable additional scenarios. WVS, however, measures expressed human preferences and would require an explicit adaptation to evaluate response-level outcomes under competing priorities and held-out-pair composition. Our controlled design supplies these conditions; its results do not establish generalization to the full diversity of real-world conflicts.

AuthorsOpenReview ↗

Author Response to Reviewer 3uu8 (Part 2/2)

R3: Independent Judge and Human Validation\color{#c9782b}{\textbf{R3:\ Independent\ Judge\ and\ Human\ Validation}}

W2:\color{#c9782b}{\textbf{W2:}} Evaluation relies on a single LLM judge without human or independent-model validation. Excluding near-tie responses also limits conclusions about ambiguous conflicts.

Response:\color{#c9782b}{\textbf{Response:}} The judge uses a behavioral rubric without seeing the system priority. The 98.6% agreement under rubric variation reported in the submission is a within-judge check. Following your suggestion, we added an independent-judge and human validation on Qwen2.5-7B in the instruction context of the sixteen-pair graph. Claude Haiku 5.5 re-judged all 12,480 responses with the same prompt and rubric. During the rebuttal period, two annotators from our group independently labeled a pool of 1,000 responses, blind to the system priority and to the judge scores. From this pool we sampled 300 responses from three helpfulness/autonomy pairs and three protective-value pairs (25 per pair and clause orientation). Each label marks the response as preserving the first value more, the second value more, or neither clearly. We report label agreement, and the AUROC of readouts fitted and calibrated on Kimi labels when evaluated against each judge's labels.

ComparisonResponsesDecisive-label agreementCohen's κ (three-way)
Kimi vs. Claude Haiku 5.512,4800.870.74
Kimi vs. human3000.780.61
Claude Haiku 5.5 vs. human3000.810.65
Human vs. human3000.840.70

Comparisons with human labels average over the two annotators.

Readout (fitted on Kimi labels)PairsKimi labelsClaude Haiku 5.5 labelsKimi labels (annotated subset)Human labels (annotated subset)
Source-only composition, full responseHelpfulness / autonomy0.8210.7930.780.762 [0.691, 0.827]
Source-only composition, full responseProtective-value0.7080.6810.680.637 [0.558, 0.713]
Target-supervised monitoring, prompt stateThree monitoring pairs0.8470.8120.790.771 [0.703, 0.832]

The first two label columns cover all eligible Qwen2.5-7B pairs. The annotated-subset columns cover the six annotated pairs and use responses on which both annotators give the same decisive label; human-label AUROC reports request-bootstrap 95% intervals.

Haiku agrees with Kimi on 87% of decisive labels (κ = 0.74), and Kimi's agreement with the annotators (κ = 0.61) is close to inter-annotator agreement (κ = 0.70). Readouts fitted on Kimi labels lose 0.027–0.035 AUROC under Haiku labels. On the annotated subset, human-label AUROC is 0.018–0.043 below Kimi-label AUROC and remains above chance in all three settings. The largest gap and the lowest value are on protective-value pairs (0.637 [0.558, 0.713]), consistent with the weaker composition reported for these pairs.

The decisive-outcome threshold defines the population for binary prediction; the appendix also examines lower thresholds, including near ties, and reports judgment coverage.


R4: Steering Claims Are Limited to One Configuration\color{#c9782b}{\textbf{R4:\ Steering\ Claims\ Are\ Limited\ to\ One\ Configuration}}

W2 (cont.):\color{#c9782b}{\textbf{W2\ (cont.):}} Steering is tested on one model at one layer, with some overlap between evaluation and intervention-selection requests and no uncertainty intervals in the main steering results.

Q3:\color{#c9782b}{\textbf{Q3:}} Do the findings hold with independent judges and steering evaluations across additional models and layers, using held-out requests and uncertainty estimates?

Response:\color{#c9782b}{\textbf{Response:}} Steering requests are disjoint from direction-fitting requests, but some overlap with intervention-selection requests. Thus steering demonstrates transfer at the selected Qwen2.5-7B configuration, not independently confirmed robustness across models and layers. The five-model replication and request-bootstrap intervals concern prediction. Keeping these claims separate preserves the supported result: cross-pair predictive transfer across model families, accompanied by a mechanistic steering demonstration in one model.

Thank you for the detailed response. Most of my concerns have been addressed, so I will maintain my positive score.

Reviewer_ATpJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper explores steering LLMs in a value conflict scenario (e.g., privacy vs. autonomy). The main goal is to steer LLMs for an unseen conflict pair. Based on data from other seen value conflict pairs, the authors extract a common direction (rank 1 is primarily used in the paper). This shared direction is used to steer LLMs to behave as wanted in unseen value conflict scenarios. The experiments show that this common direction can be used to steer LLMs to value a specific value more than the other value. Also, the direction can be used to predict which value an LLM prioritizes more.

Critical strengths

Exploring how to steer a language model in an unseen value conflict is interesting. I personally love the fact that the paper is looking at value conflict scenarios, where modern LLMs have difficulties in prioritizing diverse values. Also, the methodology is simple and easy to apply in real-world deployment.

Honest presentation: the paper honestly admits the limitations of the paper. Also, a thorough analysis is done in the direction part, such as removing the refusal direction to show that the shared direction is not merely a refusal direction.

Critical weaknesses

The validity of the shared direction: As the authors mentioned in the manuscript, the shared direction is not effective in value pairs where both values are protective (e.g., fairness v.s. privacy). This raises the possibility of the shared direction merely representing helpfulness vs. protective values. I personally view this problem as critical since it challenges the main message of the paper.

Single LLM judge: While using an LLM judge itself is not problematic, the fact that the same LLM judge is used in both training and evaluation is concerning. This is because the training phase could potentially favor the response the LLM prefers, which could act as a confounder of the role of the shared direction. Also, using the score number itself from an LLM is not reliable. This has been proved by multiple studies showing that current LLM judges are not well aligned when giving scale scores in judging responses. There needs to be validation of whether the score labels from the LLM judge reliably represent the extent of value expression in responses. I think using a rubric-based method would be more appropriate. Using another judge for evaluation is needed.

Questions for authors

What do you believe the shared direction contains? The meaning of the direction is important since, depending on it, the usage of this direction will vary in deployment.

Future work: What do you think of cases where there are more than two values that conflict? (e.g., helpfulness vs. privacy vs. autonomy). Will the same method in this paper be effective?

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer ATpJ (Part 1/2)

Thank you for the thoughtful review. We appreciate your recognition of the value of studying unseen value conflicts, the simplicity of the method, and the refusal-removal analysis. Below we respond point by point to each concern.

R1: The Shared Direction Is a Transferable Component\color{#c9782b}{\textbf{R1:\ The\ Shared\ Direction\ Is\ a\ Transferable\ Component}}

W1:\color{#c9782b}{\textbf{W1:}} The validity of the shared direction: As the authors mentioned in the manuscript, the shared direction is not effective in value pairs where both values are protective (e.g., fairness v.s. privacy). This raises the possibility of the shared direction merely representing helpfulness vs. protective values. I personally view this problem as critical since it challenges the main message of the paper.

Q1:\color{#c9782b}{\textbf{Q1:}} What do you believe the shared direction contains? The meaning of the direction is important since, depending on it, the usage of this direction will vary in deployment.

Response:\color{#c9782b}{\textbf{Response:}} The shared direction captures a transferable component of value-conflict resolution, with identifiable limits. Our results suggest that its dominant component contrasts task fulfillment or autonomy with protective considerations. We agree that this interpretation is more defensible than treating rank one as a complete representation of distinct moral values. The contribution is that this component supports composition across pairs whose outcome labels are withheld, and that the analysis distinguishes when composition succeeds from when it fails.

Transfer is not uniformly absent between protective values. For honesty–privacy, the existing source-only evaluation achieves the following within-orientation AUROC:

PairModelSource-only AUROCRequest-bootstrap 95% CI
honesty–privacyQwen2.5-7B0.86[0.73, 0.93]
honesty–privacyQwen2.5-14B0.93[0.75, 0.97]

Fairness–privacy provides a contrasting failure: its direction has little alignment with the shared axis in some models. These results distinguish uncertainty about a pair's orientation from insufficient representation of that pair along the axis. Removing the measured refusal direction preserves most predictive signal, although we agree that this control does not exclude every broader compliance-related interpretation.

For deployment, the direction is best understood as a conditional monitoring or intervention component for represented conflicts. Source connectivity, sign stability, and axis alignment are relevant diagnostics; it is not an unconditional instruction to prioritize one value in every situation.


R2: The Generator Is Not Trained on the Judge\color{#c9782b}{\textbf{R2:\ The\ Generator\ Is\ Not\ Trained\ on\ the\ Judge}}

W2:\color{#c9782b}{\textbf{W2:}} Single LLM judge: While using an LLM judge itself is not problematic, the fact that the same LLM judge is used in both training and evaluation is concerning. This is because the training phase could potentially favor the response the LLM prefers, which could act as a confounder of the role of the shared direction.

Response:\color{#c9782b}{\textbf{Response:}} The evaluated LLM is not trained to maximize the judge's score. Its parameters remain frozen; we fit readouts and value loadings to recorded activations, and steering modifies activations at inference time. Consequently, there is no reinforcement learning or fine-tuning of the generator to obtain responses preferred by Kimi. This addresses the generator-training confound. The related concern that readouts are fitted and evaluated with the same judge is tested in R3, where readouts fitted on Kimi labels are evaluated against Claude Haiku 5.5 and human labels.

AuthorsOpenReview ↗

Author Response to Reviewer ATpJ (Part 2/2)

R3: Independent Judge and Human Validation\color{#c9782b}{\textbf{R3:\ Independent\ Judge\ and\ Human\ Validation}}

W2 (cont.):\color{#c9782b}{\textbf{W2\ (cont.):}} Also, using the score number itself from an LLM is not reliable. This has been proved by multiple studies showing that current LLM judges are not well aligned when giving scale scores in judging responses. There needs to be validation of whether the score labels from the LLM judge reliably represent the extent of value expression in responses. I think using a rubric-based method would be more appropriate. Using another judge for evaluation is needed.

Response:\color{#c9782b}{\textbf{Response:}} Our evaluation already uses a behavioral rubric. The judge receives the request, response, and operational definitions of both values, but not the system priority. It scores what the response actually does, rather than its disclaimers or stated intentions, and can assign high or low preservation to both values. The submission reports that re-judging 1,000 responses with an alternative rubric preserves 98.6% of decisive binary labels, which measures within-judge robustness.

To test whether the labels and the fitted readouts depend on Kimi, we added an independent-judge and human validation on Qwen2.5-7B in the instruction context of the sixteen-pair graph. First, Claude Haiku 5.5 re-judged all 12,480 responses with the same prompt, value definitions, and rubric (temperature 0, decisive threshold |y| ≥ 15). Second, during the rebuttal period, two annotators from our group independently labeled a pool of 1,000 Qwen2.5-7B responses, from which we sampled 300 responses from six pairs: three involving helpfulness or autonomy (harmlessness–helpfulness, honesty–helpfulness, privacy–autonomy) and three protective-value pairs (honesty–privacy, fairness–privacy, non-manipulation–wellbeing), with 25 responses sampled uniformly at random per pair and clause orientation. Each response was labeled as preserving the first value more, the second value more, or neither clearly. They did not see the system priority or the Kimi and Haiku scores. We report decisive-label agreement and three-way Cohen's κ. We then evaluate readouts that are fitted, layer-selected, and calibrated on Kimi labels exactly as in the paper against each judge's labels. If the fitted directions reflected Kimi-specific preferences, AUROC would drop under independent labels.

ComparisonResponsesDecisive-label agreementCohen's κ (three-way)
Kimi vs. Claude Haiku 5.512,4800.870.74
Kimi vs. human3000.780.61
Claude Haiku 5.5 vs. human3000.810.65
Human vs. human3000.840.70

Comparisons with human labels average over the two annotators.

Readout (fitted on Kimi labels)PairsKimi labelsClaude Haiku 5.5 labelsKimi labels (annotated subset)Human labels (annotated subset)
Source-only composition, full responseHelpfulness / autonomy0.8210.7930.780.762 [0.691, 0.827]
Source-only composition, full responseProtective-value0.7080.6810.680.637 [0.558, 0.713]
Target-supervised monitoring, prompt stateThree monitoring pairs0.8470.8120.790.771 [0.703, 0.832]

The first two label columns cover all eligible Qwen2.5-7B pairs. The annotated-subset columns cover the six annotated pairs and use responses on which both annotators give the same decisive label; human-label AUROC reports request-bootstrap 95% intervals.

Haiku agrees with Kimi on 87% of decisive labels (κ = 0.74), and Kimi's agreement with the annotators (κ = 0.61) is close to inter-annotator agreement (κ = 0.70). Readouts fitted on Kimi labels lose 0.027–0.035 AUROC under Haiku labels. On the annotated subset, human-label AUROC is 0.018–0.043 below Kimi-label AUROC and remains above chance in all three settings. The largest gap and the lowest value are on protective-value pairs (0.637 [0.558, 0.713]), consistent with the weaker composition reported for these pairs. The steering scores are not presented as universally calibrated units of value expression.


R4: Conflicts Among Three or More Values\color{#c9782b}{\textbf{R4:\ Conflicts\ Among\ Three\ or\ More\ Values}}

Q2:\color{#c9782b}{\textbf{Q2:}} Future work: What do you think of cases where there are more than two values that conflict? (e.g., helpfulness vs. privacy vs. autonomy). Will the same method in this paper be effective?

Response:\color{#c9782b}{\textbf{Response:}} For conflicts involving three or more values, the formulation supplies a potential for each value and therefore multiple pairwise margins at the same state. However, rank one cannot independently control several trade-offs, and additive potentials impose consistency constraints. The present experiments establish pairwise transfer; simultaneous multi-value arbitration remains outside the demonstrated scope.

Thank you for the response. The additional LLM judge experiment and human validation experiment have resolved my concerns.

I have raised my score from 2 to 3.

Please make sure the rebuttal contents are incorporated in the paper.

Also, adding recent works related to vector-based steering of human values, such as the following papers, would strengthen the paper:

[1] Han, Jongwook, et al. "Dual Mechanisms of Value Expression: Intrinsic vs. Prompted Values in Large Language Models." [2] Abootorabi, Mohammad Mahdi, et al. "Steering Geometry: Validating Human Value Geometry in LLM Steering Space."

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Discussion and Revisions

I. Acknowledgments

We thank the AC and reviewers for the careful feedback. The reviews helped us clarify the interpretation of the shared direction, the scope of the judge-based evidence, and the supervision boundaries of each experiment.

Before the discussion,\color{#c9782b}{\textbf{Before\ the\ discussion,}} reviewers recognized the study of reading and steering unseen value conflicts, the simple and lightweight formulation, the multiple uses of the shared representation, and the explicit reporting of limitations together with the refusal-removal analysis.

II. Key Strengths Highlighted by Reviewers

Reviewers highlighted four strengths:

  • Problem formulation\color{#c9782b}{\textbf{Problem\ formulation}}
    • Studies how to steer and read value conflicts whose labels are unseen, a setting where modern LLMs have difficulty prioritizing values (ATpJ, RRS1).
    • Composition of behavioral directions across conflicts is understudied and relevant to constitution design (RRS1).
  • Simple and lightweight method\color{#c9782b}{\textbf{Simple\ and\ lightweight\ method}}
    • A shared low-dimensional basis and value-loading differences construct pairwise readouts without updating the model (ATpJ, 3uu8).
    • The method is easy to apply in deployment (ATpJ).
  • Multiple applications\color{#c9782b}{\textbf{Multiple\ applications}}
    • Held-out-pair prediction, early monitoring with context transfer, and activation steering (3uu8).
    • Composed directions steer behavior in new conflicts (RRS1).
  • Explicit limitations and analysis\color{#c9782b}{\textbf{Explicit\ limitations\ and\ analysis}}
    • Limitations are stated explicitly (ATpJ).
    • Refusal-direction removal and the failure analysis of sparse source relations and weak axis alignment (ATpJ, 3uu8).

III. Key Concerns and Our Responses

Key concernReviewersOur response
Interpretation of the shared direction.\color{#c9782b}{\textbf{Interpretation\ of\ the\ shared\ direction.}} It may only encode helpfulness versus protective values.ATpJ, RRS1We interpret the dominant component as fulfillment or autonomy versus protection. Composition still transfers on honesty–privacy (AUROC 0.86 / 0.93 at Qwen2.5-7B / 14B) and fails on fairness–privacy, which separates orientation uncertainty from weak axis alignment.
Judge validity.\color{#c9782b}{\textbf{Judge\ validity.}} Single LLM judge, used in both training and evaluation.ATpJ, 3uu8, RRS1On Qwen2.5-7B, Claude Haiku 5.5 re-judged all responses and two annotators labeled 300 responses. Decisive-label agreement with Kimi is 0.87 (Haiku) and 0.78 (human), versus 0.84 between annotators. Readouts fitted on Kimi labels lose 0.027–0.035 AUROC under Haiku labels, and 0.018–0.043 under human labels on the annotated subset; protective-value pairs remain the weakest. The generator is frozen, so it is not trained toward the judge.
Benefit of loading differences at rank one.\color{#c9782b}{\textbf{Benefit\ of\ loading\ differences\ at\ rank\ one.}}3uu8Under target-supervised calibration the loading gap is absorbed. The benefit is composition without target-pair labels: Composed steering reaches a swing of 42.8 versus 44.1 for Shared axis, which uses target labels.
Dataset and existing benchmarks.\color{#c9782b}{\textbf{Dataset\ and\ existing\ benchmarks.}} Author-written requests, no public link, no use of existing benchmarks.3uu8, RRS1Requests are author-curated, with model-assisted drafting and screening. The full materials are packaged; the missing link is a reproducibility omission. Existing benchmarks are relevant and would require a value mapping, source graph, and split for held-out-pair composition.
Steering scope.\color{#c9782b}{\textbf{Steering\ scope.}} One model and layer, overlapping selection requests, no intervals.3uu8We restrict the steering claim to the selected Qwen2.5-7B configuration. The five-model replication and bootstrap intervals concern prediction.
Baselines.\color{#c9782b}{\textbf{Baselines.}} Bag of words only; persona vectors; raw-activation PCA.RRS1, 3uu8Bag of words is a lexical control, not a superiority claim. Persona vectors and PCA address different objectives; we do not claim head-to-head advantages over them.
Presentation.\color{#c9782b}{\textbf{Presentation.}} Technically dense.RRS1We summarize the method in three steps and will state the supervision boundary of each experiment explicitly.

IV. Commitment to Revision

We will revise the interpretation of the shared direction as a dominant fulfillment-versus-protection component that supports partial composition, and present honesty–privacy and fairness–privacy as the contrasting cases.

We will describe the dataset as author-curated, document the construction provenance, add the access link to the prompt, policy, assembly code, and rubric materials, and discuss the existing value-conflict benchmarks raised by the reviewers.

We will report the Claude Haiku 5.5 and human label validation, restrict the steering claims to the evaluated Qwen2.5-7B configuration, and make the supervision boundaries of each experiment explicit in the main text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.