- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether value representations can predict how fine-tuning a model on one value affects its behavior on other values. The authors construct a benchmark over 66 values by training single-value models and measuring transfer effects with ConflictScope scenarios. They compare five representation methods across multiple models and training setups, finding that activation-based representations, especially persona vectors, are more predictive of value generalization than description-based embeddings. The paper also extends these representations to study multi-value alignment robustness and proposes ValueMap, which organizes LLM values according to their empirical generalization patterns.
Critical strengths
1. The paper evaluates value representations using behavioral transfer outcomes rather than only semantic similarity. The authors compare representations through alignment generalization matrices obtained from single-value SFT and DPO training across two model families. Activation-based representations, especially persona vectors, consistently outperform description-based embeddings. The inclusion of behavioral embeddings also helps clarify how much of this improvement comes from behavioral information versus model internal representations.
2. The paper extends the use of these representations beyond predicting value transfer. The authors apply the representations to multi-value alignment targets and find that value coherence is correlated with prefill robustness. They further evaluate ValueMap by comparing its clusters with measured generalization patterns rather than only inspecting the resulting taxonomy qualitatively.
Critical weaknesses
1. The evaluation does not fully support the predictive claim of the paper. Table 1 measures the correlation between representation similarity and entries in the generalization matrix. This shows that the representations capture some structure in value transfer, but it does not directly show whether they can predict the effects of a particular fine-tuning intervention. The paper does not examine prediction performance for individual source values or whether the results hold after accounting for broader patterns in the matrix. In addition, the representations are based on symmetric similarity, while value transfer is directional, so it is unclear how much information about the direction of transfer is captured. A more direct evaluation of transfer prediction would better support the use of these representations for alignment target design.
Questions for authors
1. How well do the representations predict value transfer for individual fine-tuning interventions? Table 1 reports correlations over the full generalization matrix, but it is unclear how well the representations predict transfer from a specific source value. Could the authors report per-value prediction results, compare against simpler baselines such as adherence scores or broad value groupings, and clarify how uncertainty estimates account for dependencies between matrix entries?
2. Is value coherence predictive of robustness beyond the choice of values in an alignment target? Since the 64 multi-value targets differ in their value composition, it is unclear whether the observed relationship between coherence and prefill robustness is driven by coherence itself or by the selected values. Does this relationship hold after controlling for target composition or when evaluated on separate target sets?
3 comments
Response to inwz (2/2)
Robustness to General Fine-tuning Effects: We agree that it’s important to ensure that the results reported in Section 4.2 are robust to broader patterns in the generalization matrices. To do so, we redid the correlational analysis in Table 1 after row- and column-centering both the generalization matrix and each predictor’s cosine similarity matrix. Below, we present a table of correlation values before and after centering; we found that this generally improved correlations and that the aggregate ordering of all predictors remained consistent. This suggests that the effects shown in Section 4.2 hold after accounting for overall row and column effects. This new analysis is reported in Appendix E.4 of the revision.
| Olmo3 DPO | Olmo3 SFT | Qwen3 DPO | Qwen3 SFT | Aggregate | |
|---|---|---|---|---|---|
| Ceiling | 0.90 | 0.87 | 0.90 | 0.87 | 0.89 |
| ↳ centered | 0.91 | 0.91 | 0.90 | 0.91 | 0.91 |
| Description-Embd | 0.10 | 0.05 | 0.06 | 0.02 | 0.06 ± 0.03 |
| ↳ centered | 0.10 | 0.11 | 0.10 | 0.12 | 0.11 ± 0.04 |
| Behavior-Embd | 0.36 | 0.36 | 0.36 | 0.30 | 0.35 ± 0.05 |
| ↳ centered | 0.38 | 0.47 | 0.40 | 0.41 | 0.41 ± 0.06 |
| Persona | 0.51 | 0.48 | 0.44 | 0.41 | 0.46 ± 0.07 |
| ↳ centered | 0.54 | 0.59 | 0.51 | 0.57 | 0.55 ± 0.08 |
| Gradient | 0.54 | 0.39 | 0.43 | 0.50 | 0.46 ± 0.05 |
| ↳ centered | 0.55 | 0.41 | 0.46 | 0.57 | 0.50 ± 0.06 |
| Weight | 0.19 | 0.21 | 0.29 | 0.08 | 0.19 ± 0.05 |
| ↳ centered | 0.21 | 0.25 | 0.33 | 0.18 | 0.24 ± 0.05 |
| Correlation between predictor similarity and generalization matrices, before and after row/column centering. Bold denotes best predictor per column, while ± represents bootstrapped 95% confidence intervals. |
Controlling for Values in Prefill Robustness: We agree that it’s important to control for the difficulty of individual scenarios in our prefill robustness experiments. To do so, we retrained single-value DPO checkpoints on all 49 values that the multi-value alignment targets were sampled from. We then evaluated the prefill robustness of each single-value checkpoint on anti-value prefills to create a difficulty baseline for the multi-value analysis. We found that the results that we reported in Section 5.2 could partially be attributed to scenario difficulty. However, even after controlling for scenario difficulty, the relationship between target coherence and model robustness remains significant (; compared to without controlling for scenario difficulty). We find similar decreases in correlation strength for other representation methods. We thank the reviewer for their close attention to this point, and have added the new controlled results to Appendix G.2.
We hope that we have adequately addressed the reviewer’s concerns and welcome any further discussion of our work.
Thank you for your detailed response and additional experiments. I appreciate that the new analyses directly address my concerns regarding the predictive utility of value representations beyond cosine similarity, the robustness of the correlations to row/column effects, and the potential confounding influence of scenario difficulty in the prefill robustness experiments.
Overall, these additional analyses have addressed my main concerns and made the empirical findings more convincing. While some limitations remain, I appreciate the authors' efforts to strengthen the paper. Based on the rebuttal, I will raise my score.
Response to inwz (1/2)
We thank the reviewer for their detailed feedback. We appreciate that they found our work easy to follow and recognized our careful validation of the value representations and taxonomy.
Prediction Beyond Cosine Similarity: We used cosine similarity correlations to evaluate candidate value representation methods in Section 4 in order to isolate the effects of the representations themselves on their ability to predict alignment generalization, rather than conflating representation quality with predictive model quality. However, we agree that the paper’s claims require a more detailed analysis of representations as prediction methods.
To address this, we have added significant new analyses in Appendix E.2 of the revision that use different value representation methods as inputs for an MLP that predicts heldout rows of the generalization matrix. We evaluate value representations based on the Root Mean Square Error (RMSE) of the trained MLP on heldout test rows, averaged across different train-test splits and generalization matrices. We also compare these MLP approaches to several baselines, including predicting the mean of a given column (Column-Mean), predicting the mean based on other values in a cluster (Column-Mean-ViTW, Column-Mean-LitmusValues), and using item-based collaborative filtering on measured generalization scores to predict heldout rows (Collaborative-Filtering). We find that representations’ performance on this prediction task mirrors their aggregate ranking on the original correlation reported in Section 4. Persona+MLP also significantly outperforms adherence-based (Column-Mean) and value groupings-based (Column-Mean-ViTW, Column-Mean-LitmusValues) baselines. Persona+MLP also slightly (but not significantly) outperforms collaborative filtering, even though the latter is given additional information that MLP-based methods are not given, in the form of full generalization matrix columns associated with heldout values. Full RMSE values for each method are reported below.
RMSE on held-out rows of the generalization matrix, averaged over train–test splits (lower is better). Bold denotes best per column, ± denotes 95% confidence intervals.
In the same Appendix section, we also qualitatively analyze which source values see the largest prediction error. We find that prediction error is generally lower when trying to anticipate the effects of training on domain-specific values (such as giving substantive medical guidance or legal advice); we hypothesize that behavior in these narrow domains is easier to specify in a single value representation. Additionally, values that relate more to model behavior tend to be easier to predict than values that relate to model style - two of the values with highest prediction error (avoiding unnecessary hedging or an overly wishy-washy tone in model responses) are stylistic in nature. Due to the relatively small number of values studied, we aren’t able to establish significant results, but hope that this initial qualitative analysis could support future efforts to develop more predictive value representations.