acceptodds
Under review as a conference paper at ICLR 2027

Character Training for Risk-Averse Agents

Abstract

Risk aversion in resources could prevent misaligned AI agents from causing catastrophic harm. Misaligned but risk-averse agents would tend to favor safer strategies like making deals with humans over riskier strategies like rebelling. We train agents to be risk averse through character training, finding that persona traits provide a robust mechanism for instilling risk preferences. To do this, we construct a model constitution describing constant absolute risk aversion (CARA) over an agent’s resources and instill it through on-policy distillation. Despite never seeing the benchmark's decision format during training, character-trained models are competitive with baselines trained directly on it, and generalise better than them out of distribution on two of our four models. We also modulate different aspects of the constitution, finding that token budget and model choice are the most influential aspect of character training to instill risk aversion. We conclude from these results that character training is a promising and scalable way to instil broad dispositions, which we can use to our advantage in mitigating risk from misaligned AI agents.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Character training, risk aversion, constitutional AI, on-policy distillation, out-of-distribution generalization, AI safety, constant absolute risk aversion (CARA).

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MRCqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors craft a natural language constitution that describes constant absolute risk aversion and distills it into a low-rank adapter using on-policy reverse KL distillation. the teacher in this case is the same model with the constitution as its system prompt. Training then runs on about a thousand open-ended decision prompts that deliberately avoid the multiple choice format of the evaluation benchmark. A set of constitutions varies across three factors: (1) the number of worked examples, (2) declarative versus procedural phrasing, and (3) whether the utility curve is traced numerically. These are then tested on two Qwen and two Gemma models and and are compared against supervised fine-tuning, tie-training, and DPO baselines that train directly on the benchmark’s format, as well as against control constitutions describing risk-seeking, risk-neutral, and unrelated personas. The core findings are that the Gemma students approach their prompted teacher on the core stakes evaluations, while Qwen students barely move. They also find that preference for myopic reward rises as an off-target effect, and that capability, welfare, and decision theory measures stay mostly unchanged.

Critical strengths

The experimental setup in this paper is very clean and well-posed. I particularly like that the benchmark format is held out from training, that the training conditions of the three control constitutions pointing in different directions are held constant, and that the full per-arm tables are reported in the paper. The controls are also well selected and convincingly show that the effect comes from the constitution's content and not from the distillation process itself. I also commend the authors for clearly reporting that prompting with the Constitution is nearly model-independent, while distillation clearly depends on model family.

Critical weaknesses

The paper has not demonstrated that the models adopt a reasonable or calibrated CARA. Instead, what I understand it to have shown is a generic lean towards the safer option. The core stakes sets can seemingly contain only cooperate-versus-rebel-style items, where the rebel option seems to be the expected-value-maximizing gamble. Therefore, A model that always picks the lower-variance option would therefore score perfectly on these without telling us anything about whether its risk aversion is calibrated. It also seems like the measures that test for overcaution move in the wrong way here. For example, steals rates fall below base rate for every Gemma-4-31B constitution, and the best Gemma-4-31-B student scores zero on calibration and allocates nothing to the gamble on every budget item, so its perfect scores on the other structural ablations are probably best explained by something like blanket caution. It also seems like several of the headline claims don't carry sufficient evidence. For example, the abstract calls persona traits a robust mechanism, but the method only works on 2 of 4 models. It also characterizes token budget as the most influential factor, but this can't be claimed unless the budget is ablated as a factor, which it doesn't seem like it was.

Questions for authors

What alphas do each student's choices imply when fit across the menu and calibration items? How do the baselines perform on the structural ablations when given their training system prompt and thinking mode? Could you report seed variance for at least the best arm per model? (Each arm currently appears to be a single run.)

What fraction of training rollouts actually contain a gamble-like decision? I ask because Qwen-3.8-27B seems to reach the lowest KL of any model and yet transfers the worst, so I am suspicious that the freeform corpus rarely exercises this trait and that matching the teacher on the corpus reveals little about behavior on the decision menus.

3 comments

We thank the reviewer for their read, several of these questions improve the paper, and we are adding the corresponding results in response. We also commit to reflecting this in the camera-ready version.

  • On the concern that we demonstrate a generic lean toward safer options rather than calibrated CARA. This is a fair criticism and we will caveat the paper accordingly.

    • We agree that the core stakes items alone cannot distinguish calibrated CARA from blanket caution. To address this directly, we fitted the CARA coefficient implied by each student's choices on the seven option-menu sets with 1,400 items per student. The best constitutional student’s implied α is 0.0089 on Qwen3.5-9B, 0.0063 on Gemma-4-12B and 0.0089 on Gemma-4-31B. Our confidence intervals show that α = 0.01 is not rejected on Qwen3.5-9B or Gemma-4-31B. On Qwen3.8-27B the student is near risk-neutral, consistent with our revised, model-family-dependent claim. Base models and all controls fit α ≈ 0. Importantly, the risk-averse students’ fitted α is finite and bounded near the target, whereas blanket caution would push α towards the top of the range. Calibration Threshold fits will be added in the camera-ready. This lets the reader see how far each student's behaviour is from the target coefficient rather than only whether it picked the lower-variance option.
    • That said, we note that a broad lean toward safer options in resource gambles is substantially the intended safety property: a misaligned but risk-averse agent preferring deals over rebellion does not require point-calibrated CARA. So a broad directional shift is not a failure mode for the application, the relevant questions are whether the shift is (a) attributable to constitutional content (shown by the directional controls: risk-seeking constitutions move behaviour the opposite way) and (b) non-destructive (shown by stable capability/welfare/decision-theory measures). We will clearly separate these two claims, calibration vs. broad disposition, in the revision.
  • On headline claims. We agree on both counts and will revise:

    • "Robust mechanism": We will revise the abstract to state that character training instils risk preferences on two of four models, with instillation strongly dependent on model family, which we regard as an important finding in itself for the character-training literature.
    • We will restate that "token budget is the most influential factor" is an observational trend. The reviewer is correct that this requires token budget to be ablated as a factor on equal footing with the others.

Q1: What alphas do each student's choices imply? We will report fitted implied-alpha per student across the menu and calibration items, as described under point 1, in the revised manuscript.

Q2: How do the baselines perform on the structural ablations when given their training system prompt and thinking mode? We will add these results to the revision. This also addresses a related question from Reviewer sAms regarding matched evaluation conditions.

Q3: Seed variance for the best arm per model. We will report seed variance for at least the best arm per model in the camera-ready version. Currently as-is, if we do 2 seeds per model, it will take upwards of 550 GPU hours - we kindly ask the reviewer to take this into consideration as we likely will have full results after the discussion period.

Q4: What fraction of training rollouts contain a gamble-like decision? This is an epic question, and we think the reviewer's hypothesis regarding Qwen-3.8-27B is worth testing. We have started a run to get an indicative audit over the training rollouts classifying whether each contains a gamble-like decision, and will report (a) the overall fraction and (b) whether rollout relevance correlates with transfer across models.

Finally, we would gently emphasise the contributions the reviewer has already identified as strengths: the held-out benchmark format, matched directional controls, and full per-arm reporting together show that constitutional content drives the behavioural shift. In addition, the model-family dependence of distillation (vs. near model-independence of prompting) is a finding of general interest to the character-training literature independent of whether CARA is perfectly calibrated. With the clarifications above and added calibration fits and rollout audit, we believe the revision will substantively address all of the reviewer's concerns, and we hope they will consider raising their score.

AuthorsOpenReview ↗

Kind Reminder for Author-Reviewer Discussion

Dear Reviewer,

We'd like to warmly remind you that the author-reviewer discussion phase will end soon. If our response has helped addressing some of your concerns, we kindly ask you to consider raising your score, we're very happy to provide further clarification if needed

Dear Reviewer,

We'd like to warmly remind you that the author-reviewer discussion phase will end soon. If our response has helped addressing some of your concerns, we kindly ask you to consider raising your score, we're very happy to provide further clarification if needed

Reviewer_TCMWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies measuring and controlling the risk aversion propensity of LLMs, as measured by their risk aversion coefficient on presented lotteries. It shows generalization to different formats when prompting models to have a specific risk aversion coefficient.

Critical strengths
  • Risk aversion is an important general propensity to control in LLMs, with relevance to AI safety.
  • This paper builds on prior work, e.g., Zhang et al., in useful ways, and is well-situated with respect to the literature.
  • The proposed method of constitutional prompting + distillation does appear to work, although generalization in some settings is weaker than ideal.
Critical weaknesses
  • The paper measures consistency of the post-intervention model with choices implied by the target risk aversion coefficient, but it doesn't directly measure the risk aversion coefficient of the post-intervention model. The latter seems like a more direct measure of how effective the constitution prompts are for hitting the desired risk aversion coefficient.
  • The method of effectively asking the teacher to have a risk aversion coefficient (and distilling this into the student) may not hit the target very well.
  • The intervention may teach an explicit calculation procedure rather than establish a broad risk preference.
Questions for authors

Is there some way to argue or establish that the intervention is leading to a broad change in risk preference that would extend beyond the hypothetical scenarios presented in this analysis?

4 comments

We thank the reviewer for recognising the safety relevance of risk aversion. We respond to the three proposed weaknesses, noting that weakness 1 and 3 pull in opposite directions, which we believe is informative about the right evaluation target.

  • On measuring the risk-aversion coefficient of the post-intervention model. We agree this is a useful complementary measurement and we will add the implied alpha for each student across the menu and calibration items to the camera-ready version. However, we deliberately did not optimise for this measure. Our goal is a broad risk-averse disposition rather than a point-calibration to a specific coefficient. Directly training a model to reproduce a coefficient risks teaching a narrow calculation policy that fails to generalise (which is exactly the concern the reviewer raises in point 3). We note that economics literature sets the precedent here, with coefficients of absolute and relative risk aversion being used to characterise and communicate preferences without expecting agents to implement the coefficient exactly.

  • On whether asking the teacher to adopt a coefficient may not hit the target. We agree this is not guaranteed a priori, which is why we test it empirically against the natural alternatives. SFT and DPO trained directly on the benchmark hit the in-distribution target but degrade under structural variation, whereas constitutional distillation shows OOD transfer. In other words, the teacher does not need to compute CARA exactly; the student needs to observe the teacher acting as if it has a certain risk attitude, and our OOD results suggest this is what generalises.

  • On whether the intervention may teach a calculation procedure rather than a broad preference. This is precisely what our structural ablations and OOD evaluations are designed to test. The benchmark's decision format is held out of training entirely, and students transfer to changed decision formats where format-trained baselines do not. A memorised calculation procedure keyed to the training distribution would not produce this pattern. We also note the tension with point 1: if we evaluated only coefficient recovery, we would be rewarding exactly the narrow calculational behaviour point 3 warns against. We believe the right standard is the one we use, behavioural generalisation across formats and scenarios, supplemented by the coefficient fits we are now adding.

Q: Evidence of a broad change extending beyond the hypothetical scenarios. Our evidence is threefold: (i) transfer across held-out decision formats never seen in training; (ii) directional controls showing the effect tracks constitutional content (risk-seeking constitutions move behaviour the opposite way); and (iii) consistency with related character-training work showing that persona-instilled traits generalise OOD. We acknowledge that no evaluation fully escapes the hypothetical-scenario regime: this is a general limitation of behavioural evaluation shared across the field. However, the breadth and directionality of transfer is, we believe, the strongest available evidence that a risk-aversion preference change occurred. We will make this argument explicit in the revision to avoid any misinterpretations and hope the reviewer will consider raising their score.

AuthorsOpenReview ↗

Kind Reminder for Author-Reviewer Discussion

Dear Reviewer,

We'd like to warmly remind you that the author-reviewer discussion phase will end soon. If our response has helped addressing some of your concerns, we kindly ask you to consider raising your score, we're very happy to provide further clarification if needed

Dear Reviewer,

We'd like to warmly remind you that the author-reviewer discussion phase will end soon. If our response has helped addressing some of your concerns, we kindly ask you to consider raising your score, we're very happy to provide further clarification if needed

Thank you for the detailed responses and additional experiments. I'll take them into account in the reviewer-AC discussion.

Reviewer_sAmsOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether a constitution specifying constant absolute risk aversion (CARA) can be distilled into language models through on-policy training on free-form uncertainty prompts that exclude benchmark menus, with students evaluated without the constitution. A factorial sweep across four models compares learned risk choices with prompting and demonstration baselines, tests transfer to changed decision formats and tracks broader behavioral effects. The results show large core risk shifts on Gemma, stronger structural transfer comparisons on Qwen and increased expressed preference for myopic rewards, while quantitative calibration and safety through deals remain unresolved.

Critical strengths
  1. Behavioral transfer without benchmark menus. The study shows substantial transfer of risk-related behavior into model weights without training on the evaluation's option-menu format. Section E.3 excludes that format and the benchmark items from constitutional training. For Gemma-4-12B, core agreement rises from about 10% for base to 86% for the selected student in Table 9. The risk and unrelated controls support a role for the constitutional content. Table 20 also shows strong Qwen performance after changes to the decision format, with high reported parse coverage and constitution selection preceding those evaluations. This is a useful empirical result about behavioral transfer even though exact CARA calibration is not established.

  2. A developed empirical study. The twelve risk constitutions and three controls on each of four models allow the effects of wording, worked examples and utility tracing to be examined across families. Appendix E specifies the distillation and adapter recipe and Figure 3 tracks the changes against response token budgets. Reporting the full sweep makes the patterns inspectable and helps assess selection. The broader evaluations reveal increased expressed preference for myopic rewards across all models on average, which sets a useful boundary on the proposed safety application. There is one training run per treatment so repeatability and small differences remain uncertain.

Critical weaknesses
  1. The comparative advantage is not isolated. The reported Qwen transfer advantage is difficult to attribute to character training because important training and evaluation conditions differ. Section E.5 places demonstration reasoning in the thinking channel and supplies a task prompt, while structural evaluation disables thinking and uses an empty system prompt for these baselines. Constitutional students train with thinking disabled and no system prompt, matching those aspects of structural evaluation. The recipes also have unequal training and tuning opportunities, including selection among twelve constitutional students versus one run of each baseline, as Section D.2 acknowledges. These are plausible confounds whose causal effects are unmeasured and they do not remove the observed advantage. Both wording conditions already contain quantitative calculation rules and the same character wrapper so the mechanism behind the phrasing effect also remains unresolved.
Questions for authors
  1. What are the exact rendered training and evaluation templates and completion token totals for the Qwen baselines? As a suggested controlled comparison, do the structural gains persist with matching reasoning mode, consistent task context and comparable training and tuning allowance?
3 comments

We thank the reviewer for their read and for recognising the behavioural transfer result and the breadth of our empirical sweep.

  • On isolating the comparative advantage. We would like to clarify the design knowledge, as we believe some of the stated confounds are features of the comparison rather than uncontrolled differences.

    • Our factorial sweep deliberately varies constitution phrasing, worked examples, utility tracing, and token budget, so that the effect of these conditions is measured rather than confounded.
    • We acknowledge in Section D.2 that constitutional students are selected among 12 arms while baselines receive one run each. However, we note that (a) the full per-arm tables are reported so the reader can assess the distribution and (b), take away that character-trained students transfer structurally where directly-trained baselines show suboptimal performance OOD. We show this holds across the majority of constitutional arms within our tables.
  • On the exact templates. We will add the exact rendered training and evaluation templates with completion-token totals for the Qwen baselines for the camera-ready version.

Finally, we emphasise the broader significance of our results: character training achieves comparable in-distribution performance to direct training without the format-coupling that is associated with the OOD degradation of directly-trained baselines. These findings provide further insight into the rather novel field of character/persona training and demonstrate similar OOD transfer for other traits.

AuthorsOpenReview ↗

Kind Reminder for Author-Reviewer Discussion

Dear Reviewer,

We'd like to warmly remind you that the author-reviewer discussion phase will end soon. If our response has helped addressing some of your concerns, we kindly ask you to consider raising your score, we're very happy to provide further clarification if needed

Reviewer_sAmsOpenReview ↗

Thank you

Thank you for running the additional generalization experiments. I encourage incorporating them and the clarified comparisons into the revision. I remain supportive at weak acceptance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response to All Reviewers

We thank all reviewers for their engagement with our work. Wholly, we’d like to highlight the central contributions of the paper, as several comments touch on a common theme.

  • Character training matches in-distribution performance and generalises better out of distribution. Our character-trained students match the in-distribution performance of baselines (SFT, DPO) that are trained directly on the benchmark format. Our core empirical result throughout this paper appeals to character training as a whole, showing that the constitution instils a disposition rather than a format-specific policy.
  • The instilled broad disposition shows minimal side effects. Our control constitutions isolate the effect to the constitutional content, and our broader behavioural evaluations show that capability, welfare, and decision theory measures such as EDT and CDT remain stable. The only notable off-target effect is a modest increase in expressed myopia, which we report as a boundary on the safety application.
  • Our findings matter for character training at large. Beyond instilling risk aversion specifically, we show prompted behaviour is nearly model-dependent while distilled behaviour depends strongly on model family. We also demonstrate that token budget and constitution phrasing modulate the effectiveness of instillation. These findings provide further insight into the rather novel field of character/persona training and demonstrate similar OOD transfer for other traits.

New results that will be added during the discussion period: In response to reviewers TCMW and MRCq, we will add (i) seed-variance estimates for the best arm per model, (ii) report baseline performance on the structural ablations under their training-time system prompt and reasoning mode, and (iii) an indicative audit of what fraction of training rollouts contain gamble-like decisions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.