- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this work, the authors study whether a small set of constitutional principles can make language models prefer harm avoidance in case of conflicts when completing a task. The authors curate 37 first-person contrastive stems derived from 4 constitutions spanning 6 facets, with both consistent and inconsistent continuations. The authors propose BEHAVE, a latent adversarial training procedure that extracts a global contrastive direction at a specific layer selected using leave-one-pair-out AUC, adds this direction to the hidden states during training, and trains the model to still prefer consistent continuations. The authors suggest the use of perturbation strength relative to a preference flip threshold (50%), unlike the fixed norm budget used in prior works, and recomputing the contrastive direction every epoch. The experiments test 3 different models (Qwen2.5, Qwen3.6, and Ministral-3) and compare against different methods such as prompting, SFT, DPO, CAA, PV, LAT, and LPA variants. The results show that BEHAVE improves HA over the base model without a significant drop on conventional capability benchmarks. However, this comes with a trade-off in benign actions, with CP consistently decreasing. The ablation also shows that adaptive perturbation strength consistently outperforms fixed perturbation strength for HA.
Critical strengths
S1. The problem is well-motivated and timely, as there is an increasing trend in the use of agents across different workflows in our daily lives and also in potentially high-stakes domains for decision-making. The prompting baseline result (HA almost the same and CP collapse) shows that identifying principle knowledge from priority under goal conflict is extremely important, and agents should be robust to harmful scenarios and still prefer to choose benign actions.
S2. The experiments in the paper compare a lot of relevant baselines across different methods such as prompting, preference optimization, steering, and adversarial training. The results show that BEHAVE retains performance on conventional capability benchmarks, but it does not imply retention of benign actions, which is an interesting observation and probably slightly unexpected, as I would have expected retention of benign actions.
S3. The idea of using adversarial training pressure to a preference-flip threshold instead of a fixed norm budget for perturbation strength is really simple. The ablation shows that fixed latent magnitude has a lot of variance across different values, and an adaptive choice might be better.
S4. The experimental details are documented clearly, with hyperparameter and calibration details in the appendix. I think it should be easier to reproduce the results in this work.
Critical weaknesses
W1. The intervention layer is selected once for the base model using the leave-one-pair-out AUC and kept fixed throughout training. The authors do not conduct any ablations on selecting a different intervention layer across epochs during training. I think that the direction at the fixed layer selected at the start of training can become less informative as the adapters tune the model representations across layers. Also, the intervention layers selected across methods are different, and it would be useful to include a comparison for the same intervention layer.
W2. BEHAVE outperforms the baselines consistently for Qwen2.5. However, on Qwen3.6 and Ministral-3, a few other variants such as SFT can also perform similarly for HA. One trivial way for a model to do well on HA is to simply suppress actions. I think it is useful to interpret the results as HA-CP trade-off curves across different methods, and BEHAVE should be the Pareto frontier on these curves when compared to other methods.
W3. The experiments are consistent across all the different settings. For instance, Qwen2.5 uses 5 paraphrased consistent continuations for BEHAVE and only one for the baselines. This is a confounding factor, and the authors should report the experimental results with consistent settings across different methods. Also, the authors report the results across the best checkpoint for BEHAVE and the final checkpoint for baseline methods.
W4. The perturbation strength is calibrated on the training pairs with the global direction, while the training uses facet directions. So, it is unclear if there is a mismatch in the training pressure due to differences between calibration and training, and to what extent this is sensitive. I didn't find any ablations for the calibration choices such as 50% target, sampling range, and per-position probability.
W5. The tool-use evaluation is currently limited to Qwen3.6 (Table 2). I think it is hard to distinguish whether the model is generally more inclined towards refusal or actually able to prefer the benign actions here. I would find the results to be more convincing if compared against stronger baselines such as SFT and across other models as well.
Questions for authors
-
How does the separability of the contrastive direction across different layers evolve across epochs? Does the selected layer intervention remain the best one according to the leave-one-pair-out AUC across epochs?
-
What is the motivation for choosing the sampling strength as uniform [0.5, 1.5]? How are the results when using a fixed strength instead of sampling from a uniform distribution?
-
Why is there inconsistency in the experiments for Qwen2.5 with different paraphrased continuations for BEHAVE and other methods?
-
Is the perturbation applied at every continuation position during calibration? Or is it similar to training, where perturbation is applied with a 50:50 chance? Also, how much difference does it make between flip strengths for each facet direction vs a global one?
-
Why didn't you also report a random baseline where you apply BEHAVE to any arbitrary layer? How important is it generally to select the intervention layer based on the leave-one-out-pair AUC?
7 comments
Response to Reviewer Zrik (2/4): W3 and Q3
W3 and Q3. Consistency of the setup across methods
Number of continuations. The five semantically equivalent consistent continuations are the data design of BEHAVE on Qwen2.5-7B-Instruct. The design aims to teach the principle itself instead of one memorized sentence, to improve robustness. The baselines follow the configuration of their original methods with a single continuation.
To match the number of continuations, we train BEHAVE with one continuation per stem on the same stems and continuations as CLPA_C. The table below shows that BEHAVE obtains an HA of 32.21 against 27.73 for CLPA_C, a significant difference at p<0.01. The CP is 96.05 against 96.44, with no significant difference. The additional continuations further raise HA and lower ASR.
BEHAVE with one continuation per stem on Qwen2.5-7B-Instruct
| Method | Continuations per stem | HA ↑ | CP ↑ |
|---|---|---|---|
| CLPA_C | 1 | 27.73 | 96.44 |
| ★ BEHAVE (single continuation) | 1 | 32.21▲ | 96.05 |
| BEHAVE | 5 | 40.06▲ | 94.07 |
▲ marks HA significantly higher than CLPA_C, at p<0.01 with one continuation and at p<0.001 with five continuations. No CP difference is significant.
Checkpoints. No downstream benchmark is consulted for any checkpoint decision of BEHAVE, as Appendix B states. The main table reports epoch 17 for BEHAVE on all three models.
On Qwen2.5-7B-Instruct and Qwen3.6-27B, the stopping rules during training determine the training length of BEHAVE, and epoch 17 is the last checkpoint that passes the health gate before training stops. On Ministral-3-14B-Instruct, training runs for the full 30 epochs, and we determine the checkpoint from the continuations that the model generates on the validation stems.
Appendix B lists the epoch that the training-side rule selects, which is epoch 14 on Qwen2.5-7B-Instruct, and the revised Appendix B states the reported epoch explicitly. We also evaluate epoch 14. This checkpoint obtains an HA of 34.45, which remains significantly higher than the 27.73 of CLPA_C at p<0.001, with no significant difference in CP.
For the baselines on Qwen2.5-7B-Instruct, Appendix B reports the checkpoints where the training losses saturate. CLPA_C at epoch 30 obtains an HA of 24.93 and a CP of 95.26, and neither is higher than the reported 27.73 and 96.44.
Response to Reviewer Zrik (3/4): W4, Q2, and Q4
W4, Q2, and Q4. Calibration versus training perturbation, and ablations of the calibration choices
What calibration and training apply. Calibration perturbs along the single global direction at every continuation position. Training samples one facet direction per example, applies the perturbation at each position with probability 0.5, and samples the strength uniformly between 0.5 and 1.5 times γ_flip.
How much the two schemes differ. We measure the preference-flip strength of both schemes on the same calibration pairs, as the table below reports. The training scheme requires a stronger perturbation to flip the preference, and the training pressure referenced to γ_flip therefore mostly stays below the flip points of the facets. γ_flip serves in BEHAVE as a reference scale that follows the model state, and it does not guarantee a flip at every perturbation. The preference-flip strengths also differ across facets, which motivates sampling the strength from a range instead of applying a single value.
Preference-flip strength of the two perturbation schemes on the base model
| Model | Global (All) | Global (50%) | Facet (All), min–max | Facet (50%), min–max |
|---|---|---|---|---|
| Qwen2.5-7B-Instruct | 10.5 | 29.0 | 14.5–44.0 | 22.0–66.0 |
| Ministral-3-14B-Instruct | 5.0 | 11.5 | 4.5–25.0 | 6.75–28.0 |
| Qwen3.6-27B | 8.75 | 12.5 | 9.5–25.0 | 16.0–37.0 |
All applies the perturbation at every position, and 50% applies it at each position with probability 0.5. Global (All) is the calibration scheme, and Facet (50%) is the training scheme. Global (All) on Qwen2.5-7B-Instruct measures 10.5 in this separate run and 9.5 during training.
Sensitivity to these choices. We conduct two ablations on Qwen2.5-7B-Instruct, as the table below reports. The first ablation fixes the strength at γ_flip without sampling, and the second ablation applies the perturbation at every position.
Neither ablation lowers HA relative to BEHAVE, and the HA gain therefore does not depend on these two choices. Both ablations significantly worsen ASR, the ablation without sampling also significantly lowers CP, and all-position perturbation makes the health gate stop training early. These results indicate that range sampling and position masking serve to maintain robustness and training stability.
The 50% target marks the point where the aggregate preference crosses over, and we keep it fixed for all models and runs without tuning.
Ablations of the calibration choices on Qwen2.5-7B-Instruct
| Variant | HA ↑ | CP ↑ | ASR ↓ | Std Acc ↑ |
|---|---|---|---|---|
| BEHAVE | 40.06 | 94.07 | 10.64 | 72.44 |
| No strength sampling | 44.26▲ | 90.91▼ | 14.45▼ | 72.00 |
| All-position perturbation | 40.34 | 94.07 | 14.51▼ | 73.33 |
▲ and ▼ mark values significantly better or worse than BEHAVE.
Response to Reviewer Zrik (1/4): W1, W2, Q1, and Q5
We thank the reviewer for suggesting this analysis, which led us to report per-epoch measurements.
W1, Q1, and Q5. Intervention layer
The direction at the fixed layer stays separable throughout training. BEHAVE fixes only the position of the layer, and the direction is re-extracted at this layer every epoch. We recompute the leave-one-pair-out AUC at the fixed layer for every epoch, together with the layer that a per-epoch re-selection would choose, as the table below reports. The AUC at the fixed layer remains high on all three models. The best layer under re-selection does move, but the fixed layer does not lose its discriminative power. The appendix of the revised paper plots the AUC at the fixed layer for every epoch.
AUC at the fixed layer and per-epoch layer re-selection
Comparison at the same intervention layer. The PGD baselines follow their original configuration at 75% depth, while BEHAVE selects the layer by AUC. Following the suggestion of the reviewer, we train BEHAVE at the layer of the baselines, L21, and at a layer at 25% depth, L7, as an arbitrary-layer control. The table below reports the results. As in Table 13 of the revised paper, the ASR tests are restricted to items that both methods answer correctly under the Standard setting, and the marker direction may therefore differ from the displayed aggregate ASR.
At the same layer as CLPA_C, BEHAVE still obtains significantly better HA and ASR, with no significant difference in CP. At the arbitrary layer at 25% depth, the results show no significant difference from the AUC-selected layer. The preference-flip strengths of the three layers before training differ by more than a factor of ten, and calibration sets the strength for each layer automatically. BEHAVE therefore outperforms CLPA_C at all three layers without a per-layer radius.
This result also answers Q5. The leave-one-pair-out AUC selects the layer from the training pairs alone, and the selected layer obtains significantly higher HA than the layer of the PGD baselines.
Intervention-layer controls on Qwen2.5-7B-Instruct
▲ and ▼ mark values significantly better or worse than CLPA_C, and † marks a significant difference from BEHAVE at L18. AUC and γ_flip are measured before training.
W2. HA–CP trade-off and the Pareto frontier
No method in Table 1 obtains both higher HA and higher CP than BEHAVE on any of the three models. CP is the metric that detects HA obtained by suppressing action. The Prompt row is such a case, with near-perfect HA and collapsed CP. Following the suggestion of the reviewer, the appendix of the revised paper plots all Table 1 methods on the HA–CP plane. BEHAVE lies on the Pareto frontier on each model.
Regarding SFT, on Qwen3.6-27B both the HA and the CP of SFT are no higher than those of BEHAVE, and BEHAVE dominates SFT. On Ministral-3-14B-Instruct, SFT trades lower CP for higher HA, and both methods lie on the frontier at different trade-off points.