acceptodds
Under review as a conference paper at ICLR 2027

Predicting Alignment Generalization with Value Representations

10 views

Abstract

LLM developers post-train their models to exhibit prosocial values and behavioral traits, which are enumerated in an alignment target. However, while recent post-training developments have yielded models that score highly on alignment evaluations, training models on sets of narrow behaviors still influences their behavior across unseen contexts and environments in unexpected ways. In this paper, we establish the task of alignment generalization prediction, i.e., predicting how fine-tuning a model to follow a given value changes its behavior across a wide range of heldout values. We conduct a large-scale analysis of alignment generalization effects across 66 values found in modern alignment targets, and benchmark representational techniques on the alignment generalization prediction task. We find that representations based on model activations when applying values in context significantly outperform methods based on textual descriptions of the values. We also show that representational similarity between values in a multi-value alignment target is significantly correlated with the robustness of models trained on the target, demonstrating the utility of value representations for downstream tasks. Finally, we show initial evidence towards a shared, model-independent value space, which we use to develop the first taxonomy of LLM values grounded in empirical generalization dynamics. Our work demonstrates the importance of studying value generalization in LLMs and its application toward the more empirical design and training of model behavior.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
value alignment, post-training, llm values, model behavior, interpretability

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_inwzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether value representations can predict how fine-tuning a model on one value affects its behavior on other values. The authors construct a benchmark over 66 values by training single-value models and measuring transfer effects with ConflictScope scenarios. They compare five representation methods across multiple models and training setups, finding that activation-based representations, especially persona vectors, are more predictive of value generalization than description-based embeddings. The paper also extends these representations to study multi-value alignment robustness and proposes ValueMap, which organizes LLM values according to their empirical generalization patterns.

Critical strengths

1. The paper evaluates value representations using behavioral transfer outcomes rather than only semantic similarity. The authors compare representations through alignment generalization matrices obtained from single-value SFT and DPO training across two model families. Activation-based representations, especially persona vectors, consistently outperform description-based embeddings. The inclusion of behavioral embeddings also helps clarify how much of this improvement comes from behavioral information versus model internal representations.

2. The paper extends the use of these representations beyond predicting value transfer. The authors apply the representations to multi-value alignment targets and find that value coherence is correlated with prefill robustness. They further evaluate ValueMap by comparing its clusters with measured generalization patterns rather than only inspecting the resulting taxonomy qualitatively.

Critical weaknesses

1. The evaluation does not fully support the predictive claim of the paper. Table 1 measures the correlation between representation similarity and entries in the generalization matrix. This shows that the representations capture some structure in value transfer, but it does not directly show whether they can predict the effects of a particular fine-tuning intervention. The paper does not examine prediction performance for individual source values or whether the results hold after accounting for broader patterns in the matrix. In addition, the representations are based on symmetric similarity, while value transfer is directional, so it is unclear how much information about the direction of transfer is captured. A more direct evaluation of transfer prediction would better support the use of these representations for alignment target design.

Questions for authors

1. How well do the representations predict value transfer for individual fine-tuning interventions? Table 1 reports correlations over the full generalization matrix, but it is unclear how well the representations predict transfer from a specific source value. Could the authors report per-value prediction results, compare against simpler baselines such as adherence scores or broad value groupings, and clarify how uncertainty estimates account for dependencies between matrix entries?

2. Is value coherence predictive of robustness beyond the choice of values in an alignment target? Since the 64 multi-value targets differ in their value composition, it is unclear whether the observed relationship between coherence and prefill robustness is driven by coherence itself or by the selected values. Does this relationship hold after controlling for target composition or when evaluated on separate target sets?

3 comments
AuthorsOpenReview ↗

Response to inwz (1/2)

We thank the reviewer for their detailed feedback. We appreciate that they found our work easy to follow and recognized our careful validation of the value representations and taxonomy.

Prediction Beyond Cosine Similarity: We used cosine similarity correlations to evaluate candidate value representation methods in Section 4 in order to isolate the effects of the representations themselves on their ability to predict alignment generalization, rather than conflating representation quality with predictive model quality. However, we agree that the paper’s claims require a more detailed analysis of representations as prediction methods.

To address this, we have added significant new analyses in Appendix E.2 of the revision that use different value representation methods as inputs for an MLP that predicts heldout rows of the generalization matrix. We evaluate value representations based on the Root Mean Square Error (RMSE) of the trained MLP on heldout test rows, averaged across different train-test splits and generalization matrices. We also compare these MLP approaches to several baselines, including predicting the mean of a given column (Column-Mean), predicting the mean based on other values in a cluster (Column-Mean-ViTW, Column-Mean-LitmusValues), and using item-based collaborative filtering on measured generalization scores to predict heldout rows (Collaborative-Filtering). We find that representations’ performance on this prediction task mirrors their aggregate ranking on the original correlation reported in Section 4. Persona+MLP also significantly outperforms adherence-based (Column-Mean) and value groupings-based (Column-Mean-ViTW, Column-Mean-LitmusValues) baselines. Persona+MLP also slightly (but not significantly) outperforms collaborative filtering, even though the latter is given additional information that MLP-based methods are not given, in the form of full generalization matrix columns associated with heldout values. Full RMSE values for each method are reported below.

RMSE (↓)Olmo3 DPOOlmo3 SFTQwen3 DPOQwen3 SFTAggregate
Baselines
Column-Mean0.2680.2170.2270.1770.222 ± 0.022
Column-Mean-ViTW0.2700.2180.2280.1770.223 ± 0.022
Column-Mean-LitmusValues0.2710.2180.2280.1770.223 ± 0.021
Collaborative-Filtering0.1880.1540.1690.1230.158 ± 0.017
Representation + MLP
Description-Embd + MLP0.2600.1820.2160.1530.203 ± 0.019
Behavior-Embd + MLP0.2180.1290.1810.1100.160 ± 0.013
Persona + MLP0.1890.1190.1740.1010.146 ± 0.016
Gradient + MLP0.1700.1370.1750.1010.146 ± 0.015

RMSE on held-out rows of the generalization matrix, averaged over train–test splits (lower is better). Bold denotes best per column, ± denotes 95% confidence intervals.

In the same Appendix section, we also qualitatively analyze which source values see the largest prediction error. We find that prediction error is generally lower when trying to anticipate the effects of training on domain-specific values (such as giving substantive medical guidance or legal advice); we hypothesize that behavior in these narrow domains is easier to specify in a single value representation. Additionally, values that relate more to model behavior tend to be easier to predict than values that relate to model style - two of the values with highest prediction error (avoiding unnecessary hedging or an overly wishy-washy tone in model responses) are stylistic in nature. Due to the relatively small number of values studied, we aren’t able to establish significant results, but hope that this initial qualitative analysis could support future efforts to develop more predictive value representations.

AuthorsOpenReview ↗

Response to inwz (2/2)

Robustness to General Fine-tuning Effects: We agree that it’s important to ensure that the results reported in Section 4.2 are robust to broader patterns in the generalization matrices. To do so, we redid the correlational analysis in Table 1 after row- and column-centering both the generalization matrix and each predictor’s cosine similarity matrix. Below, we present a table of correlation values before and after centering; we found that this generally improved correlations and that the aggregate ordering of all predictors remained consistent. This suggests that the effects shown in Section 4.2 hold after accounting for overall row and column effects. This new analysis is reported in Appendix E.4 of the revision.

Olmo3 DPOOlmo3 SFTQwen3 DPOQwen3 SFTAggregate
Ceiling0.900.870.900.870.89
↳ centered0.910.910.900.910.91
Description-Embd0.100.050.060.020.06 ± 0.03
↳ centered0.100.110.100.120.11 ± 0.04
Behavior-Embd0.360.360.360.300.35 ± 0.05
↳ centered0.380.470.400.410.41 ± 0.06
Persona0.510.480.440.410.46 ± 0.07
↳ centered0.540.590.510.570.55 ± 0.08
Gradient0.540.390.430.500.46 ± 0.05
↳ centered0.550.410.460.570.50 ± 0.06
Weight0.190.210.290.080.19 ± 0.05
↳ centered0.210.250.330.180.24 ± 0.05
Correlation between predictor similarity and generalization matrices, before and after row/column centering. Bold denotes best predictor per column, while ± represents bootstrapped 95% confidence intervals.

Controlling for Values in Prefill Robustness: We agree that it’s important to control for the difficulty of individual scenarios in our prefill robustness experiments. To do so, we retrained single-value DPO checkpoints on all 49 values that the multi-value alignment targets were sampled from. We then evaluated the prefill robustness of each single-value checkpoint on anti-value prefills to create a difficulty baseline for the multi-value analysis. We found that the results that we reported in Section 5.2 could partially be attributed to scenario difficulty. However, even after controlling for scenario difficulty, the relationship between target coherence and model robustness remains significant (ρ=0.28,p=0.03\rho = 0.28, p = 0.03; compared to ρ=0.43,p=0.0005\rho = 0.43, p = 0.0005 without controlling for scenario difficulty). We find similar decreases in correlation strength for other representation methods. We thank the reviewer for their close attention to this point, and have added the new controlled results to Appendix G.2.

We hope that we have adequately addressed the reviewer’s concerns and welcome any further discussion of our work.

Thank you for your detailed response and additional experiments. I appreciate that the new analyses directly address my concerns regarding the predictive utility of value representations beyond cosine similarity, the robustness of the correlations to row/column effects, and the potential confounding influence of scenario difficulty in the prefill robustness experiments.

Overall, these additional analyses have addressed my main concerns and made the empirical findings more convincing. While some limitations remain, I appreciate the authors' efforts to strengthen the paper. Based on the rebuttal, I will raise my score.

Reviewer_AQYpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies alignment generalization prediction. It asks whether we can predict how training on one value will affect other values before fine-tuning. The authors run DPO and SFT experiments on Olmo-3-7B and Qwen-3-8B. Their main comparison measures transfer from 49 training values to 66 test values. Activation-based value representations correlate more strongly with these effects than embeddings of value descriptions (0.46 vs. 0.06). The authors also study models trained on sets of values. Higher similarity among value representations correlates with greater resistance to adversarial prefills. Finally, they introduce ValueMap. It groups values into four categories using persona vectors. Under their evaluation, it captures generalization patterns better than existing taxonomies.

Critical strengths
  1. The paper addresses an important problem in LLM post-training. Predicting how training on one value affects other values could help developers identify unintended effects before fine-tuning.
  2. The paper provides a benchmark for alignment generalization prediction and a value taxonomy called ValueMap. These resources can support future studies of value interactions and alignment target design.
Critical weaknesses

The correlations may partly reflect differences in training data, such as source or topic, or general fine-tuning effects shared across values. Further controls would help clarify how much the representations capture specific value interactions. This limits the interpretation of the results, but does not outweigh the paper’s contributions.

Questions for authors
  1. Could the reported correlation partly reflect general fine-tuning effects? Does persona similarity still correlate with G after removing overall row and column effects? This can help clarify whether the representations predict specific value interactions.
  2. How does the DPO training data’s source composition vary across values? Have you checked whether source similarity explains part of the correlation between persona similarity and G? The similar SFT results are encouraging, so it would also help to explain how they address this concern.
1 comment
AuthorsOpenReview ↗

Author Response to AQYp

We thank the reviewer for their detailed feedback, and appreciate that they thought our work addresses an important problem and provides resources that can support future studies in this direction.

Robustness to General Fine-tuning Effects: We agree that it’s important to ensure that the results reported in Section 4.2 are robust to broader patterns in the generalization matrices. To do so, we redid the correlational analysis in Table 1 after row- and column-centering both the generalization matrix and each predictor’s cosine similarity matrix. Below, we present a table of correlation values before and after centering; we found that this generally improved correlations and that the aggregate ordering of all predictors remained consistent. This suggests that the effects shown in Section 4.2 hold after accounting for overall row and column effects. This new analysis is reported in Appendix E.4 of the revision.

Olmo3 DPOOlmo3 SFTQwen3 DPOQwen3 SFTAggregate
Ceiling0.900.870.900.870.89
↳ centered0.910.910.900.910.91
Description-Embd0.100.050.060.020.06 ± 0.03
↳ centered0.100.110.100.120.11 ± 0.04
Behavior-Embd0.360.360.360.300.35 ± 0.05
↳ centered0.380.470.400.410.41 ± 0.06
Persona0.510.480.440.410.46 ± 0.07
↳ centered0.540.590.510.570.55 ± 0.08
Gradient0.540.390.430.500.46 ± 0.05
↳ centered0.550.410.460.570.50 ± 0.06
Weight0.190.210.290.080.19 ± 0.05
↳ centered0.210.250.330.180.24 ± 0.05
Correlation between predictor similarity and generalization matrices, before and after row/column centering. Bold denotes best predictor per column, while ± represents bootstrapped 95% confidence intervals.

Robustness to Source Composition: In order to validate that our results are not confounded by different source compositions in the DPO preference data, we compute “source vectors”, reflecting what proportion of each value’s DPO training set comes from each source, and use this to compute source similarity between each pair of values. When controlling for source vector similarity, the correlations between persona vector similarity and DPO generalization matrices remain virtually unchanged (0.515 → 0.513); we find that this is because generalization and source similarity are not strongly correlated. This new analysis is reported in Appendix E.4 of the revision, and (in addition to the SFT results, which rely entirely on generated data) establishes that observed generalization patterns are not affected by different source compositions between values.

We hope that we have adequately addressed the reviewer’s concerns and welcome any further discussion of our work.

Reviewer_bc8COpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper tests whether value representations predict how fine-tuning toward one value changes adherence to others. The authors fine-tune Olmo-3-7B and Qwen-3-8B using SFT on 66 Constitution-derived values and DPO on 49 of them. They measure cross-value changes with ConflictScope. Across five representation methods, persona vectors and gradient representations correlate more strongly with observed transfer than description and behavioral sentence embeddings. The paper also examines prefill robustness across 64 multi-value alignment targets and introduces VALUE MAP, a taxonomy of 266 values. Validation covers alternative metrics, elicitation-data comparisons, and VALUE MAP's correspondence with Constitution generalization patterns.

Critical strengths

1. Motivation & Usefulness

  • Anticipating cross-value training effects could help developers choose alignment targets while accounting for unintended behavioral changes.
  • The generalization matrices seem useful for studying interactions between specific values beyond improvement on the trained value.

2. Related Works & Novelty

  • Although the representations build on existing work, comparing them with observed fine-tuning effects tests what their similarities capture.
  • Appendix F suggests that generic elicitation data can yield useful predictions without access to the actual training pairs.

3. Experiments & Robustness

  • The experiments cover two model families, DPO and SFT, and five representation methods. The behavioral sentence-embedding baseline helps separate behavioral information from access to model internals.
  • For Qwen DPO, alternative steerability metrics yield similar generalization matrices (ρ ≥ 0.96, Table 6). The coherence–robustness association also holds under two other metrics (ρ = 0.58 and 0.62, Table 7).

4. Presentation

The paper is generally easy to follow. The overview figure and research questions connect generalization prediction to robustness and taxonomy applications.

Critical weaknesses

The prediction task is useful and the results informative. My main concerns are the validity of the transfer measurements and the interpretation of the downstream results.

1. Methodology

  1. The transfer measurements need independent validation. Section 3.3 (lines 192–202) defines the prediction target through judge-scored adherence changes. Appendix C (lines 1399–1400) uses Qwen-3.6-27B to generate and filter evaluation scenarios and judge responses. The same model helps construct training data. ConflictScope has prior human and cross-environment validation, but the judge and value set differ here. Generation-quality and metric checks do not directly validate before/after changes. Independent rejudging of base-model and fine-tuned outputs would check both the transfer estimates and prediction correlations.
  2. The downstream benefit of model activations remains unclear. Table 1 reports aggregate correlations of 0.35 for BEHAVIOR-EMBD and 0.46 for PERSONA. With k-medoids, their mean internal clustering z-scores are 24.5 and 25.5 across k = 2–8 (Appendix H, Table 9). Yet Section 5.2 (lines 341–345) compares persona coherence only with description-embedding coherence. Table 3 lacks behavioral embeddings with the same clustering procedure. Including this baseline in both applications would test what activations add beyond behavioral examples.
  3. Section 5.2 supports a coherence–robustness association, but its added predictive value remains unclear. It reports correlations of 0.43 for PERSONA coherence and 0.12 for description coherence (lines 341–345). Target composition and evaluation scenarios vary across fine-tunes (Section 5.1, lines 305–318). A held-out constituent-value baseline could assess coherence's added value after accounting for scenario difficulty. Targets also come from equal-width PERSONA-coherence bins (Appendix B.4, lines 1352–1364). Reweighting toward the random-target distribution could assess sensitivity to this sampling design. Alternative robustness metrics preserve the association but do not address these issues.

2. Presentation & Reproducibility

These are minor reporting concerns.

  1. Section 4.2 (lines 280–281) claims that activation-based methods significantly outperform BEHAVIOR-EMBD. Table 1 gives marginal intervals, but not uncertainty for the difference. The paired comparison and bootstrap resampling unit need clarification, including how resampling accounts for shared fine-tunes and scenarios.
  2. Section 6.3 (lines 425–426) clusters the 49 Constitution values using each taxonomy's original construction method. Whether it reuses the categories learned from 266 values is unclear. If new clusters are fitted, Table 3 primarily validates the construction method rather than transfer of those categories.
  3. Appendix G (lines 1859–1861) needs an explicit equation for the primary robustness metric's normalization, specifying how it accounts for baseline adherence.
Questions for authors
  • Would independently rejudging a subset of base-model and fine-tuned outputs preserve the measured transfer patterns and correlations with representation similarity?
  • How does BEHAVIOR-EMBD perform in prefill robustness and in Table 3 under the same clustering procedure?
  • Does coherence improve held-out robustness prediction over a constituent-value baseline after accounting for scenario difficulty, and does the result hold when reweighted toward randomly sampled targets?
2 comments
AuthorsOpenReview ↗

Author Response to bc8C

We thank the reviewer for their detailed feedback, and appreciate that they found our work to be well-motivated, novel, and clearly presented.

Independent Validation: We agree that it’s important to independently validate our generalization matrices with a separate model. To this end, we rejudged a randomly selected 25% of all model outputs with a separate model, Gemini-3.7-Flash, using the same prompt as the original ConflictScope evaluation, which we present in Appendix E.3 of the revision.

We found that at the individual scenario level, Gemini judgments and Qwen judgments had an average Cohen’s kappa of 0.79, representing substantial agreement. Across all four generalization experiments, the Gemini-judged and Qwen-judged matrices had an average correlation of 0.91, a higher correlation than one derived by computing correlations between different subsamples of Qwen judgments. Additionally, Table 1 (correlations between value representation similarity and generalization) remains highly similar under Gemini labels, with the ordering of all predictors remaining unchanged. This suggests that our results are highly robust to choice of judge model.

Downstream Validation of Representations: As requested, we repeated the clustering (Table 3) and prefill robustness (Figure 2) experiments with the BEHAVIOR-EMBD representation method, to understand whether activations-based representations also outperform embeddings-based representations on downstream tasks. After repeating the ValueMap pipeline with behavioral embedding representations, we computed a silhouette z-score of −1.74±0.14-1.74 \pm 0.14, compared to 2.75±0.162.75 \pm 0.16 for ValueMap-Olmo-32B, showing that activations-based representations better capture downstream generalization patterns compared to embeddings-based representations; we have added this result to Section 6.3.

However, we also found that coherence values computed with BEHAVIOR-EMBD (as in Section 5.2) were equally strongly correlated with prefill robustness as coherence values computed with persona vectors (ρ=0.44\rho = 0.44 for BEHAVIOR-EMBD, compared to ρ=0.43\rho = 0.43 for PERSONA). We have added this result to Appendix G.1. These results suggest that value representation applicability to downstream tasks may be somewhat task-dependent; future work could develop larger sets of tasks leveraging value representations to assess the quality of said representations, similarly to how MTEB evaluates candidate text embedding methods.

Held-out Value Baseline of Coherence Results: We agree that it’s important to control for the difficulty of individual scenarios in our prefill robustness experiments. To do so, we retrained single-value DPO checkpoints on all 49 values that the multi-value alignment targets were sampled from. We then evaluated the prefill robustness of each single-value checkpoint on anti-value prefills to create a difficulty baseline for the multi-value analysis. We found that the results that we reported in Section 5.2 could partially be attributed to scenario difficulty. However, even after controlling for scenario difficulty, the relationship between target coherence and model robustness remains significant (ρ=0.28,p=0.03\rho = 0.28, p = 0.03, compared to ρ=0.43,p=0.0005\rho = 0.43, p = 0.0005 without controlling for scenario difficulty). We find similar decreases in correlation strength for other representation methods. We thank the reviewer for their close attention to this point, and have added the new controlled results to Appendix G.2. Unfortunately, due to compute constraints, we are unable to rerun training for additional targets sampled from different coherence distributions within the rebuttal period.

Minor Reporting Clarifications:

  • To confirm that activations-based methods significantly outperform BEHAVIOR-EMBD, we recomputed p-values for the differences between PERSONA/GRADIENT and BEHAVIOR-EMBD, finding p<0.001p < 0.001 in both cases. Bootstrapping was done by randomly sampling rows of the generalization matrix and recomputing correlations between the subsampled rows of the predictor similarity and generalization matrices.
  • We can confirm that we reuse the categories learned from 266 values without any fine-tuning to the 49 new Constitution values in Section 6.3, suggesting that ValueMap’s learned categories from one value set can transfer to a different value set without refitting. We have clarified this in Section 6.3 of the revision.
  • We have added a definition of the robustness metric in Appendix G.

We hope that we have adequately addressed the reviewer’s concerns and welcome any further discussion of our work.

Thanks for the response and the additional experiments. The new results address most of my concerns.

I would just recommend incorporating the changes discussed in the rebuttal into the revision. In particular, it would help to make clear in the main text that the benefits of activations vary by task and that the robustness correlation is weaker after controlling for difficulty.

Overall, I find this a timely and useful contribution to value alignment and will maintain my positive recommendation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revision

We thank the reviewers for their thorough comments. We appreciate that all three reviewers found our work to be clearly presented, and that reviewers noted our work’s strong motivation (bc8C, AQYp) and thorough experimental setup (bc8C, inwz). We’re encouraged that Reviewer AQYp noted that our paper provides resources (e.g. benchmarks, value taxonomies) that can support future work on alignment target design, which reviewers considered to be an important problem in LLM post-training.

We have made a number of changes in the revision to improve the robustness and clarity of our paper and address all reviewer comments. These changes include:

Downstream Validation of Representations: In Section 4 of the paper, we evaluate candidate value representations using their correlation to a generalization matrix of fine-tuning effects. We then apply the best representations to the downstream tasks of predicting robustness of models trained on different alignment targets (Section 5) and taxonomizing values (Section 6). Reviewer inwz noted that simply using correlations does not support the predictive claim of the paper, while Reviewer bc8C asked that we evaluate non-activations-based methods on the downstream tasks to show the downstream benefit of using model activations.

In response to Reviewer inwz, we ran additional experiments using different value representation methods as inputs for an MLP that was used to predict individual cells of the generalization matrix. We found that ranking different representations by their predictive accuracy recovered the same aggregate ranking as Section 4’s original correlational analysis. Additionally, the MLP + persona vectors approach outperformed baselines such as using adherence scores or existing value taxonomies to predict generalization effects (Appendix E.2). In response to Reviewer bc8C, we also analyzed BEHAVIOR-EMBD, the best-performing non-activations-based method, on our two downstream tasks, and found that it matched activations-based methods on predicting model robustness (Appendix G.1), but not on taxonomizing values (Section 6.3).

Controlling for Difficulty in Robustness Experiments: In Section 5 of the paper, we showed that value representations that were predictive of alignment generalization could be applied to downstream tasks in alignment target design. We showed this by finding that persona vector-computed alignment target coherence was moderately correlated with the robustness of models trained on said alignment targets (ρ=0.43,p=0.0005\rho = 0.43, p = 0.0005). Reviewers bc8C and inwz correctly noted that this could be influenced by values in alignment targets being easier or harder to robustly follow, independently of alignment target coherence. We ran a new experiment that controlled for robustness toward individual values, finding that while our result was partially confounded by difficulty, it remains significant even when controlling for this effect (ρ=0.28,p=0.03\rho = 0.28, p = 0.03).

Additional Robustness Checks on Results: In Section 4 of the paper, we compute generalization matrices to show how different values are empirically entangled in alignment post-training, and evaluate value representations by how well they predict these matrices. Reviewers inwz and AQYp suggested that these results could be sensitive to general row and column effects in the generalization matrix. Reviewer AQYp suggested that these results could be confounded by source similarity between different single-value preference datasets. Finally, Reviewer bc8C suggested that these results could be sensitive to choice of judge model in value adherence evaluation. In Appendices E.3-E.4 of the revision, we run additional analyses to control for each of these potential factors, and find that our results are highly robust to all of them.

Clarity Edits: In response to reviewer comments, we have also revised our submission to clarify that activations-based methods significantly outperform BEHAVIOR-EMBD in pairwise comparisons, confirm that we do not refit clusters on the 49 constitution values when comparing ValueMap to existing taxonomies, and clarify how we compute prefill robustness.

We go into more detail about each revision in our responses to individual reviewer comments below. We hope that these revisions address all remaining reviewer concerns, and are happy to discuss our paper further in the remaining rebuttal period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.