- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors craft a natural language constitution that describes constant absolute risk aversion and distills it into a low-rank adapter using on-policy reverse KL distillation. the teacher in this case is the same model with the constitution as its system prompt. Training then runs on about a thousand open-ended decision prompts that deliberately avoid the multiple choice format of the evaluation benchmark. A set of constitutions varies across three factors: (1) the number of worked examples, (2) declarative versus procedural phrasing, and (3) whether the utility curve is traced numerically. These are then tested on two Qwen and two Gemma models and and are compared against supervised fine-tuning, tie-training, and DPO baselines that train directly on the benchmark’s format, as well as against control constitutions describing risk-seeking, risk-neutral, and unrelated personas. The core findings are that the Gemma students approach their prompted teacher on the core stakes evaluations, while Qwen students barely move. They also find that preference for myopic reward rises as an off-target effect, and that capability, welfare, and decision theory measures stay mostly unchanged.
Critical strengths
The experimental setup in this paper is very clean and well-posed. I particularly like that the benchmark format is held out from training, that the training conditions of the three control constitutions pointing in different directions are held constant, and that the full per-arm tables are reported in the paper. The controls are also well selected and convincingly show that the effect comes from the constitution's content and not from the distillation process itself. I also commend the authors for clearly reporting that prompting with the Constitution is nearly model-independent, while distillation clearly depends on model family.
Critical weaknesses
The paper has not demonstrated that the models adopt a reasonable or calibrated CARA. Instead, what I understand it to have shown is a generic lean towards the safer option. The core stakes sets can seemingly contain only cooperate-versus-rebel-style items, where the rebel option seems to be the expected-value-maximizing gamble. Therefore, A model that always picks the lower-variance option would therefore score perfectly on these without telling us anything about whether its risk aversion is calibrated. It also seems like the measures that test for overcaution move in the wrong way here. For example, steals rates fall below base rate for every Gemma-4-31B constitution, and the best Gemma-4-31-B student scores zero on calibration and allocates nothing to the gamble on every budget item, so its perfect scores on the other structural ablations are probably best explained by something like blanket caution. It also seems like several of the headline claims don't carry sufficient evidence. For example, the abstract calls persona traits a robust mechanism, but the method only works on 2 of 4 models. It also characterizes token budget as the most influential factor, but this can't be claimed unless the budget is ablated as a factor, which it doesn't seem like it was.
Questions for authors
What alphas do each student's choices imply when fit across the menu and calibration items? How do the baselines perform on the structural ablations when given their training system prompt and thinking mode? Could you report seed variance for at least the best arm per model? (Each arm currently appears to be a single run.)
What fraction of training rollouts actually contain a gamble-like decision? I ask because Qwen-3.8-27B seems to reach the lowest KL of any model and yet transfers the worst, so I am suspicious that the freeform corpus rarely exercises this trait and that matching the teacher on the corpus reveals little about behavior on the decision menus.
3 comments
Kind Reminder for Author-Reviewer Discussion
Dear Reviewer,
We'd like to warmly remind you that the author-reviewer discussion phase will end soon. If our response has helped addressing some of your concerns, we kindly ask you to consider raising your score, we're very happy to provide further clarification if needed
Reminder
Dear Reviewer,
We'd like to warmly remind you that the author-reviewer discussion phase will end soon. If our response has helped addressing some of your concerns, we kindly ask you to consider raising your score, we're very happy to provide further clarification if needed
We thank the reviewer for their read, several of these questions improve the paper, and we are adding the corresponding results in response. We also commit to reflecting this in the camera-ready version.
On the concern that we demonstrate a generic lean toward safer options rather than calibrated CARA. This is a fair criticism and we will caveat the paper accordingly.
On headline claims. We agree on both counts and will revise:
Q1: What alphas do each student's choices imply? We will report fitted implied-alpha per student across the menu and calibration items, as described under point 1, in the revised manuscript.
Q2: How do the baselines perform on the structural ablations when given their training system prompt and thinking mode? We will add these results to the revision. This also addresses a related question from Reviewer sAms regarding matched evaluation conditions.
Q3: Seed variance for the best arm per model. We will report seed variance for at least the best arm per model in the camera-ready version. Currently as-is, if we do 2 seeds per model, it will take upwards of 550 GPU hours - we kindly ask the reviewer to take this into consideration as we likely will have full results after the discussion period.
Q4: What fraction of training rollouts contain a gamble-like decision? This is an epic question, and we think the reviewer's hypothesis regarding Qwen-3.8-27B is worth testing. We have started a run to get an indicative audit over the training rollouts classifying whether each contains a gamble-like decision, and will report (a) the overall fraction and (b) whether rollout relevance correlates with transfer across models.
Finally, we would gently emphasise the contributions the reviewer has already identified as strengths: the held-out benchmark format, matched directional controls, and full per-arm reporting together show that constitutional content drives the behavioural shift. In addition, the model-family dependence of distillation (vs. near model-independence of prompting) is a finding of general interest to the character-training literature independent of whether CARA is perfectly calibrated. With the clarifications above and added calibration fits and rollout audit, we believe the revision will substantively address all of the reviewer's concerns, and we hope they will consider raising their score.