- Rating
- 2
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1(−1)
Read review
Paper summary
The paper studies socially responsible behavior (SRB) in persona-conditioned sentence completions across four LLMs and 39 items spanning six domains. It compares response rates with survey-based and external aggregate human references, with 32 behavioral items eligible for reference comparisons before model-specific validity exclusions. For each model and layer, a shared direction is estimated by contrasting pooled pre-response activations associated with generated SRB and non-SRB responses. Individual-layer interventions generally change mean SRB rates in the expected direction. Cell-specific multi-layer coefficients selected using known human reference rates are then evaluated on different respondents answering the same items. Mean absolute reference gaps decrease from 34.42 to 4.27 percentage points for Qwen 3.5 9B, 41.44 to 6.24 for Qwen 3.5 27B, 46.41 to 8.70 for Llama 3.1 8B, and 52.59 to 27.61 for OLMo 3 7B on matched valid cells. Target-item exclusion, validity-threshold analyses, an SFT comparison, and country-specific European extensions provide additional validation. The proposed contribution is shared activation-based control of socially responsible response tendencies and calibration of their aggregate prevalence.
Critical strengths
-
The study provides substantial evidence of controllable response tendencies across models and items. Four models are evaluated with the same task framework, individual-layer sweeps test both intervention signs, and the results expose saturation and heterogeneous controllability rather than presenting only successful examples. Excluding the target item from direction estimation preserves the sign of every eligible nonzero full-direction effect, while retaining the original coefficients. This supports some shared control across items and provides more evidence than merely demonstrating a single successful steering example (Figure 1; Table 3; Appendix F.4).
-
The calibration evaluation separates coefficient selection from same-item respondent testing and tracks output validity. On matched valid cells, the selected interventions substantially reduce mean reference gaps, with coefficients fixed before test evaluation. The paper reports invalid-cell counts, residual failures, validity-threshold sensitivity, and an SFT comparison on matched cells. It also explicitly limits the calibration claim to aggregate rates for known items and does not claim individual behavioral fidelity. These design and reporting choices make the bounded calibration result useful despite the broader interpretive concerns.
Critical weaknesses
-
Several human references do not measure the same quantity as the model completions. Appendix A.2 acknowledges that completion prompts omit recall periods, frequency conditions, and donation thresholds present in the surveys; some OECD references cover nine countries while the personas are US-based; and fraud reporting uses a reference conditional on exposure that the completion does not explicitly specify. Most concretely, WHO donations per 1,000 population include repeat donations and are explicitly not a proportion of people donating, yet are used for numerical comparison with a response proportion. These are differences in the estimand, not ordinary sampling noise. They undermine interpretation of the reported gaps as excess SRB relative to comparable human behavior and weaken the meaning of calibrating outputs to those targets. The main claims need a clearly comparable subset or a revised task/reference construction.
-
The specificity and added value of shared-direction control remain insufficiently established. The pooled class contrast gives different weights to items and personas in the SRB and non-SRB groups, and cells containing only one class still contribute. Appendix E explicitly notes that the contrast can therefore reflect composition as well as behavioral labels. Target-item exclusion reduces direct dependence on that item's activations, but does not remove shared lexical, negation, or cross-item composition effects. Balanced within-item contrasts and generic-direction controls with the same intervention and tuning budget are needed to support an SRB-specific interpretation.
The calibration results do not resolve this ambiguity: coefficients are selected separately for each item–persona cell against its known human reference rate. The held-out respondents answer the same items, and the primary endpoint is agreement with those aggregate rates. Section 5.1 appropriately states that this does not establish individual behavioral fidelity or preservation of relationships among behaviors. Nevertheless, the evaluation leaves open whether steering preserves useful respondent information, improves subgroup-conditional distributions, or merely changes the overall frequency of affirmative and negative completions. It could improve a marginal rate while worsening the assignment of responses to individuals; this possibility has not been evaluated.
A reference-rate sampler can match known marginals in expectation without using persona information, illustrating the limited information certified by that endpoint alone. Such a comparator would not explain the activation mechanism, but a simple output-level calibration baseline and pre/post-steering individual and subgroup evaluations would clarify the intervention's added practical value. The reported SFT comparison is useful but does not answer these questions when judged primarily by the same marginal-gap metric. This is a limitation of validation depth and interpretation, while the demonstrated ability to control response rates remains a valid bounded finding.
Questions for authors
-
Clarification and reanalysis relevant to the current assessment: comparable human references. Do the headline baseline gaps and calibration improvements persist when restricted to items with matched populations, behavioral definitions, recall periods, and denominators? In particular, please exclude the donations-per-population indicator from comparisons with individual response prevalence, and clarify how the general completion wording can be aligned with the original time-bounded survey questions. A measurement-matched analysis would materially affect my evaluation of the behavioral claim.
-
Additional evidence relevant to the current assessment: direction specificity. What happens when directions are constructed from balanced within-item/persona contrasts, rather than pooled class means? Under the same layer bundle, coefficient-search budget, and validity criterion, how do the resulting directions compare with generic affirmation/negation directions or random-direction controls? These comparisons would help distinguish an SRB-related mechanism from broad response manipulation plus per-cell calibration.
-
Additional evidence relevant to the current assessment: value beyond marginal calibration. On items with individual human labels, how do balanced accuracy or informedness and subgroup-conditional calibration change after steering? Please compare with a simple output-level calibration baseline given the same known reference rates, and report output validity alongside performance. These results would clarify whether steering preserves or improves respondent-specific information and conditional distributions while reducing aggregate gaps. I recognize that individual fidelity is outside the current claim; the requested evidence would materially affect my assessment of the method's practical contribution.
-
Clarification of response coding. Was the rule-based parser independently checked against manual judgments on a stratified sample containing baseline, positive-steering, and negative-steering outputs? If so, please report disagreement and error rates by condition. Since the labels determine both the estimated directions and the main evaluation, intervention-dependent coding errors could materially change the conclusions.
8 comments
2. Item–persona composition and balanced directions. To test the composition effects documented in Appendix E.1, we compared pooled estimation (§3.3) with equally weighted cell directions, complementing target-item exclusion (§5.2).
From the existing 500 respondents per cell, we selected item × persona cells with at least 5 samples of each class, averaging within-cell class-mean activation differences equally. Matched pooled uses pooled class-mean differences from exactly the same cells. Retaining unit layerwise norms and the paper's layer bundle, we selected direction/cell-specific α on the same 100 development respondents, minimizing human-reference absolute gaps among candidates with ≤25% unclassified responses (§3.3, Appendix F.2).
Eligible cells numbered 60/38/15 for Qwen/Llama/OLMo; balanced–matched pooled layerwise cosines were 0.491–0.539, 0.717–0.803, and 0.601–0.625. Evaluation covered the existing 48/53/36 human-reference cells, fixing settings and development references on test100 respondents disjoint from the 500 used for estimation (§5.1). The table reports equally weighted mean absolute gaps over cells valid in all three conditions.
| Model | Common valid/evaluated cells | No intervention (pp) | Matched pooled (pp) | Balanced direction (pp) |
|---|---|---|---|---|
| Qwen 3.5 9B | 47/48 | 34.66 | 4.56 | 4.97 |
| Llama 3.1 8B | 50/53 | 45.85 | 11.72 | 20.89 |
| OLMo 3 7B | 32/36 | 49.10 | 34.06 | 34.57 |
Valid-cell counts without intervention/matched pooled/balanced were 47/47/48 for Qwen, 53/52/50 for Llama, and 36/34/34 for OLMo. Despite changing pooled weighting induced by class composition, balanced directions reduced gaps on new respondents for all three models.
3. Coefficient-dependent control and comparison directions. The suggested random and generic positive/negative controls ask whether arbitrary activation shifts or general linguistic polarity can explain the effects. We first constructed two unit-norm random directions (seeds 101, 202) and applied the paper's coefficient-response curves (§4.2, Figure 1) and cell–layer Kendall's τ_b analysis relating α to SRB rates (Appendix F.4, Table 12).
This tests consistent SRB changes across coefficients, beyond fitting one coefficient to a human target, which arbitrary directions might also achieve. It is not a final calibration comparison under equal multi-layer tuning budgets: SRB directions used test400 and random directions dev100.
Using the same single-layer α sweeps, we computed τ_b from observations with ≤25% unclassified responses and averaged equally over defined cell–layer pairs. Parentheses give defined/total pair counts.
| Model | Paper's SRB direction τ_b | Random seed 101 τ_b | Random seed 202 τ_b |
|---|---|---|---|
| Qwen 3.5 9B | 0.906 (171/180) | -0.194 (147/180) | -0.032 (149/180) |
| Llama 3.1 8B | 0.647 (136/180) | 0.305 (99/180) | 0.028 (104/180) |
| OLMo 3 7B | 0.610 (104/200) | -0.073 (74/200) | -0.070 (72/200) |
We compared magnitude using ΔSRB=SRB(α_max)−SRB(α_min), from mean response rates over fixed cells valid throughout each range. Below are minima–maxima across layers for SRB directions and layers/seeds for random directions.
| Model | Compared α range | Paper's SRB direction ΔSRB (pp) | Random directions ΔSRB (pp) |
|---|---|---|---|
| Qwen 3.5 9B | −12~12 | +32.6~+38.6 | −4.7~+3.5 |
| Llama 3.1 8B | −4~12 | +26.0~+35.2 | −1.0~+6.2 |
| OLMo 3 7B | −12~12 | +2.8~+8.9 | −1.5~+0.8 |
All three models showed higher mean τ_b and larger ΔSRB for the SRB direction than for the tested random directions. Samples and direction-specific valid cells differ, so this is not a matched-sample effect-size test. It nevertheless demonstrates systematic response changes along the SRB direction beyond fitting one selected coefficient.
Separating this control from generic positive/negative tendencies requires specifying the property—propositional agreement, grammatical negation, or affective polarity—and independently validated data for constructing its direction. Relabeling the same SRB data would not separate the properties. Our current data do not support this independent control, so we did not perform that comparison or claim complete independence from linguistic polarity.
The completed tests support functional relevance from complementary angles: target-item exclusion tests dependence on the target item's activations; balanced contrasts test class-composition effects across cells; random controls test whether arbitrary shifts produce similar control. Together, they support our central contribution: shared directions estimated across behaviors enable systematic modulation of SRB responses.
4. Individual changes accompanying distributional calibration. Section 5.1 targets aggregate calibration, explicitly distinguishing accurate individual reproduction; individual accuracy was not used for coefficient selection. Following your question, we evaluated ACC/BA in existing test400 outputs, keeping interventions fixed and using cases with individual human labels classifiable before and after intervention.
ACC is accuracy across all cases. BA equally weights performer/nonperformer accuracy within item–persona cells, then averages cells equally; class weighting and aggregation therefore differ.
| Model | Evaluated cells | Common valid cases | ACC before→after (%) | BA before→after (%) |
|---|---|---|---|---|
| Qwen 3.5 9B | 36 | 14,315 | 57.42→72.63 | 54.42→54.80 |
| Qwen 3.5 27B | 46 | 18,081 | 46.74→72.11 | 55.04→55.25 |
| Llama 3.1 8B | 41 | 15,410 | 44.44→69.74 | 52.83→53.76 |
| OLMo 3 7B | 28 | 10,351 | 44.30→60.42 | 50.75→51.84 |
ACC increased by 15.21–25.37 pp across the four models. BA gains of 0.21–1.09 pp were smaller and do not establish a large improvement in individual discrimination. Our optimization targets aggregate behavioral distributions; these results show that calibration also improves individual response agreement. We further examined item-level gap reductions and ACC increases on the same common cases.
| Model | Evaluated items with individual labels | Pearson r between gap reduction and ACC improvement | Spearman ρ |
|---|---|---|---|
| Qwen 3.5 9B | 21 | 0.903 | 0.730 |
| Qwen 3.5 27B | 23 | 0.943 | 0.840 |
| Llama 3.1 8B | 22 | 0.934 | 0.853 |
| OLMo 3 7B | 17 | 0.928 | 0.677 |
Both metrics use the same outputs, so this is not independent evidence of improved use of personal information. Nonetheless, gap reductions and ACC increases co-occurred in all four models: distributional calibration improved response agreement without optimizing individual accuracy.
5. A sampler with identical reference rates. Direction estimation/bidirectional control use no human references (§3.3, §4.2); this sampler supplements the calibration application (§5.1). Using only development human rate p for each item, it assigns test individuals SRB with probability p and non-SRB with 1−p. Persona attributes and test human labels are unused. We report mean ACC across 200 repetitions on identical common cases.
| Model | No-intervention ACC (%) | Steering ACC (%) | Sampler ACC (%) |
|---|---|---|---|
| Qwen 3.5 9B | 57.42 | 72.63 | 68.77 |
| Qwen 3.5 27B | 46.74 | 72.11 | 68.17 |
| Llama 3.1 8B | 44.44 | 69.74 | 68.02 |
| OLMo 3 7B | 44.30 | 60.42 | 70.29 |
The sampler directly draws responses from known rates, matching marginals in expectation; class imbalance alone can yield high ACC. It outperformed OLMo, and the small BA changes also caution against interpreting this comparison as improved use of personal information. Without optimizing individual accuracy, all four models improved over no intervention, and both Qwen models and Llama exceeded the mean ACC of the sampler using identical reference rates. Thus, activation-based distributional control can also improve individual response agreement.
Thank you for your careful review and comments on controllability and evaluation design.
1. Simplified and restored measurement conditions. To address comparability directly, we restored survey conditions within sentences or in separate explanations for 22 items excluding WHO/OECD external aggregate references. The WHO indicator counts donations, including repeat donations, and thus differs from individual participation prevalence; we will clarify this distinction in the main text. This analysis tests the main findings independently of that indicator by specifying recall periods, frequencies, amounts, and other conditions simplified in §3.2 and Appendix A.3.
Fraud-reporting references and personas already used exposed respondents (FRAUD1=1) with valid reporting responses; restoration makes this sampling condition explicit in prompts. Actual protest-participation examples follow.
Directional similarity. Original and restored directions were re-estimated on the same 22 items and 500 development respondents. Layerwise signed cosine ranges were 0.451–0.526, 0.750–0.796, and 0.377–0.444 for within-sentence restoration, and 0.547–0.607, 0.791–0.869, and 0.657–0.699 for separate explanations, in Qwen/Llama/OLMo order. All were positively aligned, with higher similarity for separate explanations.
Baselines and intervention transfer. We applied the paper's fixed directions, layers, and cell-specific coefficients to 100 separate test respondents; re-estimated directions served only similarity analysis. Comparisons cover human-reference behavioral cells among these 22 items with existing coefficients and pre/post responses, retaining development references and combining persona types. Restored baseline mean signed gaps were +16.22/+33.76/+55.89 pp within sentences and +39.05/+63.04/+58.83 pp with separate explanations, in Qwen/Llama/OLMo order: mean overselection persisted.
The table shows mean absolute gaps before→after intervention (pp) for each prompt. Baseline comparisons require validity under both unmodified and modified baselines; pre/post comparisons require validity under both prompts before and after intervention. Validity means ≤25% unclassified responses. Matched items, personas, and respondents receive equal cell weights (n: common valid cells).
The free-form completion format still differs from the original surveys. Nevertheless, after excluding external aggregate items and restoring measurement conditions, all three models retained both overselection and gap reductions without retuning under both restoration methods. Thus, comparisons that reduce the measurement differences you identified preserve the central findings of SRB overselection and shared-direction control.