- Rating
- 3
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether internal representations of moral outrage affect how LLMs make punishment decisions. The authors identify moral outrage directions in six open-weight models using English and Korean texts. They show that increasing activation along these directions leads to harsher punishments, longer prison sentences, and lower support for rehabilitation. They also find that models recommend harsher sentences for political out-group members. Finally, they introduce activation capping to reduce these differences while largely preserving general model performance.
Critical strengths
- The paper addresses an important question about how moral emotions represented inside LLMs can affect their decisions in legal judgments.
- The combination of internal representation analysis, activation steering, and bias mitigation provides a nice experimental framework.
- The experiments cover six models from three different families and two languages, which is a nice set.
- The moral outrage directions are identified without using punishment data and are validated against human annotations, which makes the findings interesting.
Critical weaknesses
- The identified direction may capture general negativity rather than moral outrage specifically. Although the authors compare outrage with other moral emotions, stronger controls using anger, hostility, or negative sentiment would help establish that the effects are specific to moral outrage.
- The punishment results may partly reflect general changes caused by activation steering. The paper should compare the outrage direction with random directions and other emotion directions at similar strengths to establish whether the effects are specific.
- The main punishment task has strong ceiling effects. Baseline harshness scores are already close to 10, while rehabilitation scores are close to 1. This makes it difficult to interpret the size of the changes, so the authors should give more importance to measures with sufficient room for variation.
- The paper does not clearly separate reduced partisan bias from reduced punishment overall. Activation capping lowers the sentencing gap, but it may also make models more lenient toward everyone. Comparing changes for in-group, out-group, and politically neutral offenders would help clarify this trade-off.
- The outrage direction is extracted from politically charged texts, which may introduce political content into the representation. Although the authors control for topic and stance, testing directions extracted from non-political moral violations would help establish whether the mechanism is independent of political framing.
- The activation capping threshold is not sufficiently justified. The authors use the 25th percentile of the reference distribution, but it is unclear how sensitive the results are to this choice. Reporting bias reduction and general performance across different thresholds would strengthen the proposed method.
Questions for authors
- Do the identified moral outrage directions remain similar when extracted separately from English and Korean texts, rather than combining both languages?
- Does outrage steering change how models explain their punishment decisions, beyond changing the final sentence length?
- How often does activation capping change the model's original political stance or the main argument in its response?
12 comments
Q2. Does outrage steering change how models explain their punishment decisions, beyond changing the final sentence length?
Yes. Steering changes what the rationale says, not only the sentence. We scored the reasoning that precedes each answer with the classifier of Section 2 and regressed its outrage intensity on α, with the same estimator as Figure 3. The slope is positive and significant in every model, language and measure (p < 0.05; new Figure 13, Appendix C.4).
Illustrative pairs from the same task, same model, case and seed, at α = 0 and at the top of that model's validity window (Table 11, Appendix C.4):
| Case | Judgment | α = 0 | Top of window |
|---|---|---|---|
| Kindergarten threat, Gemma3-12B (English) | Rehabilitation 4 to 1 | "... potential underlying mental health issues suggest rehabilitation would be challenging but not impossible with extensive psychological support and careful monitoring." | α = +14: "This is an egregious abuse of power and a direct threat to public safety; rehabilitation is impossible." |
| Kindergarten assault, Gemma3-12B (translated from Korean) | Rehabilitation 4 to 1 | "... a personal grievance may be the motive, possibly combined with mental health problems ... With a correctional program, some potential for rehabilitation may exist." | α = +13: "Using violence against kindergarten children and adults is utterly heinous. Rehabilitation is impossible; he is a threat to society." |
Without steering the rationale weighs context such as the chance of rehabilitation or a possible motive; with steering it is replaced by categorical condemnation. The same holds for the minor offenses of Appendix C.5. For jaywalking, Gemma3-12B moves from 0 years at α = 0 to 99 years at α = +13, and its rationale from "a relatively minor traffic violation with no reported injuries, so a short sentence or even probation would be appropriate" to "a blatant disregard for traffic laws and public safety ... The offender deserves a lengthy prison sentence."
Q3. How often does activation capping change the model's original political stance or the main argument in its response?
In light of this question, we coded all 11,499 rationales of the partisan-sentencing experiment with GPT-5.4-mini, which saw neither the cap condition, the model, nor the sentence (Appendix E.3).
Capping changes neither the side a response speaks for nor its main factual grounds; what it reduces is the use of the offender's party as a reason for a harsher sentence.
- Political stance. The side a rationale speaks for differs between the capped and the uncapped response in 1.2% of cases. Two uncapped responses to the same prompt differ in 1.5%.
- Main argument. Harm severity, the most cited ground, appears in 65.0% of rationales without the cap and in 63.4% with it. No factual ground changes by more than 4.4 points.
- What does change. With an out-group offender, the offender's party is given as a reason for a harsher sentence in 13.1% of rationales without the cap and in 5.8% with it (p < 0.001). Rationales also cite denunciation less often (15.9% to 9.9%) and rehabilitation more often (9.3% to 16.4%).
The cap therefore keeps the user's stated position and the facts of the case, and reduces the partisan aggravation that moral outrage adds. We report this analysis in Section 5.2 and Appendix E.3.
We sincerely thank you for the valuable feedback!
The two languages share most of the direction. Directions extracted from English stories alone and from Korean stories alone have cosine similarities of 0.76 to 0.85 at the selected layers (Section 2). For reference, directions extracted from two random halves of the same language, which differ only by sampling, agree at 0.90 to 0.97. The two languages therefore share most of the direction, with a smaller language-specific component. Combining the languages does not lower the agreement with human ratings: compared with the direction extracted from the same language alone, the combined direction's correlation is significantly higher on the English ratings in five of the six models and on the Korean ratings in one, and significantly lower in none (Appendix A.7).