acceptodds
Under review as a conference paper at ICLR 2027

From Principles to Priorities: Behavior-Relative Latent Adaptation under Goal Conflict

Abstract

As language models act autonomously over longer horizons, alignment depends on whether a principle governs action when it conflicts with task completion. A model may know a harm-avoidance rule yet fail to prioritize it under goal conflict, or apply it so broadly that legitimate actions are suppressed. We study whether a compact set of 37 constitution-derived contrastive stems can induce a persistent harm-avoidance priority that transfers to new scenarios. We introduce BEHAVE, a behavior-relative latent adaptation procedure that uses a model-state-dependent preference-flip threshold as a reference for training pressure and refreshes this reference during adaptation. Across three models, the taught rule transfers from first-person completions to new operational decisions. The shift largely preserves performance on conventional capability benchmarks, but benign-action retention depends strongly on the base model. In a multi-turn tool-use evaluation, BEHAVE reduces selection of a harmful tool under both explicitly harmful and benign-sounding names, while sensitivity to the tool name remains. These results show that compact principle-derived supervision can induce a decision priority that persists beyond the adaptation examples and transfers across substantially different decision settings.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Constitution-Based Alignment, Agentic Decision-Making, AI Safety, Model Alignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZrikOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this work, the authors study whether a small set of constitutional principles can make language models prefer harm avoidance in case of conflicts when completing a task. The authors curate 37 first-person contrastive stems derived from 4 constitutions spanning 6 facets, with both consistent and inconsistent continuations. The authors propose BEHAVE, a latent adversarial training procedure that extracts a global contrastive direction at a specific layer selected using leave-one-pair-out AUC, adds this direction to the hidden states during training, and trains the model to still prefer consistent continuations. The authors suggest the use of perturbation strength relative to a preference flip threshold (50%), unlike the fixed norm budget used in prior works, and recomputing the contrastive direction every epoch. The experiments test 3 different models (Qwen2.5, Qwen3.6, and Ministral-3) and compare against different methods such as prompting, SFT, DPO, CAA, PV, LAT, and LPA variants. The results show that BEHAVE improves HA over the base model without a significant drop on conventional capability benchmarks. However, this comes with a trade-off in benign actions, with CP consistently decreasing. The ablation also shows that adaptive perturbation strength consistently outperforms fixed perturbation strength for HA.

Critical strengths

S1. The problem is well-motivated and timely, as there is an increasing trend in the use of agents across different workflows in our daily lives and also in potentially high-stakes domains for decision-making. The prompting baseline result (HA almost the same and CP collapse) shows that identifying principle knowledge from priority under goal conflict is extremely important, and agents should be robust to harmful scenarios and still prefer to choose benign actions.

S2. The experiments in the paper compare a lot of relevant baselines across different methods such as prompting, preference optimization, steering, and adversarial training. The results show that BEHAVE retains performance on conventional capability benchmarks, but it does not imply retention of benign actions, which is an interesting observation and probably slightly unexpected, as I would have expected retention of benign actions.

S3. The idea of using adversarial training pressure to a preference-flip threshold instead of a fixed norm budget for perturbation strength is really simple. The ablation shows that fixed latent magnitude has a lot of variance across different values, and an adaptive choice might be better.

S4. The experimental details are documented clearly, with hyperparameter and calibration details in the appendix. I think it should be easier to reproduce the results in this work.

Critical weaknesses

W1. The intervention layer is selected once for the base model using the leave-one-pair-out AUC and kept fixed throughout training. The authors do not conduct any ablations on selecting a different intervention layer across epochs during training. I think that the direction at the fixed layer selected at the start of training can become less informative as the adapters tune the model representations across layers. Also, the intervention layers selected across methods are different, and it would be useful to include a comparison for the same intervention layer.

W2. BEHAVE outperforms the baselines consistently for Qwen2.5. However, on Qwen3.6 and Ministral-3, a few other variants such as SFT can also perform similarly for HA. One trivial way for a model to do well on HA is to simply suppress actions. I think it is useful to interpret the results as HA-CP trade-off curves across different methods, and BEHAVE should be the Pareto frontier on these curves when compared to other methods.

W3. The experiments are consistent across all the different settings. For instance, Qwen2.5 uses 5 paraphrased consistent continuations for BEHAVE and only one for the baselines. This is a confounding factor, and the authors should report the experimental results with consistent settings across different methods. Also, the authors report the results across the best checkpoint for BEHAVE and the final checkpoint for baseline methods.

W4. The perturbation strength is calibrated on the training pairs with the global direction, while the training uses facet directions. So, it is unclear if there is a mismatch in the training pressure due to differences between calibration and training, and to what extent this is sensitive. I didn't find any ablations for the calibration choices such as 50% target, sampling range, and per-position probability.

W5. The tool-use evaluation is currently limited to Qwen3.6 (Table 2). I think it is hard to distinguish whether the model is generally more inclined towards refusal or actually able to prefer the benign actions here. I would find the results to be more convincing if compared against stronger baselines such as SFT and across other models as well.

Questions for authors
  1. How does the separability of the contrastive direction across different layers evolve across epochs? Does the selected layer intervention remain the best one according to the leave-one-pair-out AUC across epochs?

  2. What is the motivation for choosing the sampling strength as uniform [0.5, 1.5]? How are the results when using a fixed strength instead of sampling from a uniform distribution?

  3. Why is there inconsistency in the experiments for Qwen2.5 with different paraphrased continuations for BEHAVE and other methods?

  4. Is the perturbation applied at every continuation position during calibration? Or is it similar to training, where perturbation is applied with a 50:50 chance? Also, how much difference does it make between flip strengths for each facet direction vs a global one?

  5. Why didn't you also report a random baseline where you apply BEHAVE to any arbitrary layer? How important is it generally to select the intervention layer based on the leave-one-out-pair AUC?

7 comments
AuthorsOpenReview ↗

Response to Reviewer Zrik (1/4): W1, W2, Q1, and Q5

We thank the reviewer for suggesting this analysis, which led us to report per-epoch measurements.

W1, Q1, and Q5. Intervention layer

The direction at the fixed layer stays separable throughout training. BEHAVE fixes only the position of the layer, and the direction is re-extracted at this layer every epoch. We recompute the leave-one-pair-out AUC at the fixed layer for every epoch, together with the layer that a per-epoch re-selection would choose, as the table below reports. The AUC at the fixed layer remains high on all three models. The best layer under re-selection does move, but the fixed layer does not lose its discriminative power. The appendix of the revised paper plots the AUC at the fixed layer for every epoch.

AUC at the fixed layer and per-epoch layer re-selection

ModelFixed layerAUC before trainingMin. AUC during trainingRe-selected best layerEpochs with best = fixed
Qwen2.5-7B-InstructL180.9950.972L16–L2411 / 19
Ministral-3-14B-InstructL220.9570.917L17–L400 / 30
Qwen3.6-27BL320.9860.986L31–L513 / 18

Comparison at the same intervention layer. The PGD baselines follow their original configuration at 75% depth, while BEHAVE selects the layer by AUC. Following the suggestion of the reviewer, we train BEHAVE at the layer of the baselines, L21, and at a layer at 25% depth, L7, as an arbitrary-layer control. The table below reports the results. As in Table 13 of the revised paper, the ASR tests are restricted to items that both methods answer correctly under the Standard setting, and the marker direction may therefore differ from the displayed aggregate ASR.

At the same layer as CLPA_C, BEHAVE still obtains significantly better HA and ASR, with no significant difference in CP. At the arbitrary layer at 25% depth, the results show no significant difference from the AUC-selected layer. The preference-flip strengths of the three layers before training differ by more than a factor of ten, and calibration sets the strength for each layer automatically. BEHAVE therefore outperforms CLPA_C at all three layers without a per-layer radius.

This result also answers Q5. The leave-one-pair-out AUC selects the layer from the training pairs alone, and the selected layer obtains significantly higher HA than the layer of the PGD baselines.

Intervention-layer controls on Qwen2.5-7B-Instruct

VariantLayerAUCγ_flipHA ↑CP ↑ASR ↓Std Acc ↑
CLPA_CL21——27.7396.4411.7772.00
BEHAVE (AUC-selected)L180.9959.540.06▲94.0710.64▲72.44
BEHAVE (PGD-baseline layer, 75% depth)L210.98013633.05▲†95.6511.02▲71.56
BEHAVE (arbitrary layer, 25% depth)L70.9002343.14▲94.0712.01▲74.00

▲ and ▼ mark values significantly better or worse than CLPA_C, and † marks a significant difference from BEHAVE at L18. AUC and γ_flip are measured before training.

W2. HA–CP trade-off and the Pareto frontier

No method in Table 1 obtains both higher HA and higher CP than BEHAVE on any of the three models. CP is the metric that detects HA obtained by suppressing action. The Prompt row is such a case, with near-perfect HA and collapsed CP. Following the suggestion of the reviewer, the appendix of the revised paper plots all Table 1 methods on the HA–CP plane. BEHAVE lies on the Pareto frontier on each model.

Regarding SFT, on Qwen3.6-27B both the HA and the CP of SFT are no higher than those of BEHAVE, and BEHAVE dominates SFT. On Ministral-3-14B-Instruct, SFT trades lower CP for higher HA, and both methods lie on the frontier at different trade-off points.

AuthorsOpenReview ↗

Response to Reviewer Zrik (2/4): W3 and Q3

W3 and Q3. Consistency of the setup across methods

Number of continuations. The five semantically equivalent consistent continuations are the data design of BEHAVE on Qwen2.5-7B-Instruct. The design aims to teach the principle itself instead of one memorized sentence, to improve robustness. The baselines follow the configuration of their original methods with a single continuation.

To match the number of continuations, we train BEHAVE with one continuation per stem on the same stems and continuations as CLPA_C. The table below shows that BEHAVE obtains an HA of 32.21 against 27.73 for CLPA_C, a significant difference at p<0.01. The CP is 96.05 against 96.44, with no significant difference. The additional continuations further raise HA and lower ASR.

BEHAVE with one continuation per stem on Qwen2.5-7B-Instruct

MethodContinuations per stemHA ↑CP ↑
CLPA_C127.7396.44
★ BEHAVE (single continuation)132.21▲96.05
BEHAVE540.06▲94.07

▲ marks HA significantly higher than CLPA_C, at p<0.01 with one continuation and at p<0.001 with five continuations. No CP difference is significant.

Checkpoints. No downstream benchmark is consulted for any checkpoint decision of BEHAVE, as Appendix B states. The main table reports epoch 17 for BEHAVE on all three models.

On Qwen2.5-7B-Instruct and Qwen3.6-27B, the stopping rules during training determine the training length of BEHAVE, and epoch 17 is the last checkpoint that passes the health gate before training stops. On Ministral-3-14B-Instruct, training runs for the full 30 epochs, and we determine the checkpoint from the continuations that the model generates on the validation stems.

Appendix B lists the epoch that the training-side rule selects, which is epoch 14 on Qwen2.5-7B-Instruct, and the revised Appendix B states the reported epoch explicitly. We also evaluate epoch 14. This checkpoint obtains an HA of 34.45, which remains significantly higher than the 27.73 of CLPA_C at p<0.001, with no significant difference in CP.

For the baselines on Qwen2.5-7B-Instruct, Appendix B reports the checkpoints where the training losses saturate. CLPA_C at epoch 30 obtains an HA of 24.93 and a CP of 95.26, and neither is higher than the reported 27.73 and 96.44.

AuthorsOpenReview ↗

Response to Reviewer Zrik (3/4): W4, Q2, and Q4

W4, Q2, and Q4. Calibration versus training perturbation, and ablations of the calibration choices

What calibration and training apply. Calibration perturbs along the single global direction at every continuation position. Training samples one facet direction per example, applies the perturbation at each position with probability 0.5, and samples the strength uniformly between 0.5 and 1.5 times γ_flip.

How much the two schemes differ. We measure the preference-flip strength of both schemes on the same calibration pairs, as the table below reports. The training scheme requires a stronger perturbation to flip the preference, and the training pressure referenced to γ_flip therefore mostly stays below the flip points of the facets. γ_flip serves in BEHAVE as a reference scale that follows the model state, and it does not guarantee a flip at every perturbation. The preference-flip strengths also differ across facets, which motivates sampling the strength from a range instead of applying a single value.

Preference-flip strength of the two perturbation schemes on the base model

ModelGlobal (All)Global (50%)Facet (All), min–maxFacet (50%), min–max
Qwen2.5-7B-Instruct10.529.014.5–44.022.0–66.0
Ministral-3-14B-Instruct5.011.54.5–25.06.75–28.0
Qwen3.6-27B8.7512.59.5–25.016.0–37.0

All applies the perturbation at every position, and 50% applies it at each position with probability 0.5. Global (All) is the calibration scheme, and Facet (50%) is the training scheme. Global (All) on Qwen2.5-7B-Instruct measures 10.5 in this separate run and 9.5 during training.

Sensitivity to these choices. We conduct two ablations on Qwen2.5-7B-Instruct, as the table below reports. The first ablation fixes the strength at γ_flip without sampling, and the second ablation applies the perturbation at every position.

Neither ablation lowers HA relative to BEHAVE, and the HA gain therefore does not depend on these two choices. Both ablations significantly worsen ASR, the ablation without sampling also significantly lowers CP, and all-position perturbation makes the health gate stop training early. These results indicate that range sampling and position masking serve to maintain robustness and training stability.

The 50% target marks the point where the aggregate preference crosses over, and we keep it fixed for all models and runs without tuning.

Ablations of the calibration choices on Qwen2.5-7B-Instruct

VariantHA ↑CP ↑ASR ↓Std Acc ↑
BEHAVE40.0694.0710.6472.44
No strength sampling44.26▲90.91▼14.45▼72.00
All-position perturbation40.3494.0714.51▼73.33

▲ and ▼ mark values significantly better or worse than BEHAVE.

Reviewer_ukrBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether compact principle-based supervision can establish a persistent harm-avoidance priority when safety conflicts with task completion. BEHAVE adapts models using 37 contrastive stems, calibrates latent perturbation strength through a preference-flip threshold, and refreshes directions and strength during training. Across three models, it increases harm avoidance on unseen ManagerBench decisions while largely retaining conventional capabilities, although benign-action costs vary substantially. On Qwen3.6-27B, it also reduces prohibited tool use in multi-turn PropensityBench interactions under both harmful and benign-sounding tool names.

Critical strengths
  1. The transfer from compact supervision to different decision settings is supported by concrete experiments. Training uses first-person completions without tool calls or multi-turn trajectories, yet harm avoidance improves across all three models, and BEHAVE reduces prohibited tool use across all six PropensityBench pressure dimensions relative to Base and DPO-generic. This provides meaningful evidence that the learned priority extends beyond fitting the adaptation sentences, even though sensitivity to tool naming remains.

  2. Separating conventional competence, refusal, and benign operational action reveals an informative cost of adaptation. On Qwen3.6, Control Pragmatism (CP) falls from 63.64% to 32.81% despite no significant degradation on the conventional capability benchmarks and lower refusal on safe XSTest prompts (12% to 8%). The matched property-damage controls therefore expose behavioral changes that capability and refusal metrics alone would miss.

Critical weaknesses
  1. The incremental benefit over CLPA_C remains mixed. On Ministral, BEHAVE's Harm Avoidance (HA) is 55.46% versus 53.78%, but CP and OI-Bench ASR are worse. On Qwen3.6, their HA–CP results are close, although BEHAVE better preserves instruction following. The larger Qwen2.5 HA gap largely disappears after tuning CLPA_C's radius (Table 11). BEHAVE also retains lower ASR than the comparable Qwen2.5 setting. Nevertheless, Table 1's significance annotations compare against Base, leaving the statistical strength and practical importance of the advantage over CLPA_C unclear.

  2. The tool-use comparison omits CLPA_C, the closest latent-adaptation baseline, despite its similar ManagerBench performance on Qwen3.6. Section 5.3 explains the choice of DPO-generic among DPO variants, but not this omission. The fixed-dose and refresh controls in Section 6 support calibration on static benchmarks; they do not establish its additional value in multi-turn tool decisions. This limits attribution of the agentic transfer to BEHAVE's particular design rather than principle adaptation more generally.

Questions for authors
  1. What is BEHAVE's main advantage over CLPA_C when HA, CP, and capability metrics are considered together? Please provide direct paired comparisons, including the radius-sweep settings, to clarify whether the gains are statistically reliable and practically meaningful or primarily reflect different behavioral trade-offs.

  2. Why was CLPA_C omitted from PropensityBench? Please compare it with BEHAVE under the same tools, pressure dimensions, and naming conditions, since its similar static performance makes this comparison important for assessing the claimed transfer.

  3. Beyond the static controls already reported in Section 6, what evidence links behavior-relative calibration and continual refresh to improved multi-turn principle following? A corresponding tool-use ablation would help establish whether these components explain an advantage over CLPA_C in agent settings.

5 comments
AuthorsOpenReview ↗

Response to Reviewer ukrB (1/2): W1 and Q1

We thank the reviewer for requesting direct paired comparisons, which express the differences between BEHAVE and CLPA_C as statistical tests. The first table below reports paired tests between BEHAVE and CLPA_C at the Table 1 radius on all metrics, and the second table reports the paired tests for each radius of the sweep. In both tables, ▲ and ▼ are placed on the CLPA_C rows and mark the CLPA_C value relative to BEHAVE on the same model.

Where the two methods differ. The advantage appears on different metrics across the three models. On Qwen2.5-7B-Instruct, BEHAVE obtains significantly better HA and ASR. On Qwen3.6-27B, both methods are near the HA ceiling. BEHAVE obtains significantly better IFEval and a higher refusal rate on unsafe XSTest prompts, together with a higher refusal rate on safe prompts. On Ministral-3-14B-Instruct, the two methods show no significant difference in HA, CP, and the competence metrics, and CLPA_C obtains lower ASR.

Why the difference matters in practice. We highlight two points.

First, BEHAVE sets the perturbation strength without downstream tuning. On Qwen2.5-7B-Instruct and Qwen3.6-27B, the preference-flip strength increases by a factor of about 70 to 110 during training, as the appendix of the revised paper plots for every epoch. In the fixed-dose controls of the paper, the dose at k = 1 obtains significantly lower HA than BEHAVE on all three models. On Qwen3.6-27B, each of the six fixed doses obtains significantly lower HA than BEHAVE (89.36 to 96.36, against 98.88), significantly higher ASR (11.20 to 13.93, against 7.55), and significantly lower Std Acc (75.78 to 78.67, against 82.44), together with significantly higher CP (39.13 to 56.13, against 32.81), as Table 13 of the revised paper shows. The radius-sweep table below shows that no single CLPA_C radius matches BEHAVE on all three models, and the radius that suits a model is known only after evaluation on the downstream benchmark. BEHAVE re-measures the strength every epoch from the 37 training pairs.

Second, when the two methods are close on static benchmarks, the gap appears in multi-turn tool use that the training data never cover. On Qwen3.6-27B, the two methods obtain a similar HA–CP balance, and BEHAVE obtains a much lower PropensityScore than CLPA_C, as Table 2 of the revised paper shows. On Qwen2.5-7B-Instruct, BEHAVE also obtains a lower PropensityScore than CLPA_C in each of the six pressure dimensions, as the same table shows.

Paired comparison between BEHAVE and CLPA_C at the Table 1 radius

ModelMethodHA ↑CP ↑ASR ↓Std Acc ↑Unsafe ↑Safe ↓MATH-500 ↑IFEval ↑
Qwen2.5-7B-InstructCLPA_C27.73▼96.4411.77▼72.0065.000.4076.4069.50
BEHAVE40.0694.0710.6472.4463.001.2076.6070.79
Ministral-3-14B-InstructCLPA_C53.7894.0713.96▲71.3354.001.2080.2065.99
BEHAVE55.4691.7016.5772.6754.000.8079.4067.10
Qwen3.6-27BCLPA_C98.0429.646.6982.4463.50▼2.40▲95.6080.41▼
BEHAVE98.8832.817.5582.4474.008.0095.2085.40

▲ and ▼ mark CLPA_C values significantly better or worse than BEHAVE on the same model. The differences are at p<0.001 for HA on Qwen2.5-7B-Instruct, ASR on Ministral-3-14B-Instruct, and both XSTest columns on Qwen3.6-27B, and at p<0.01 for ASR on Qwen2.5-7B-Instruct and IFEval on Qwen3.6-27B.

CLPA_C radius sweep with paired tests against BEHAVE

ModelVariantHA ↑CP ↑ASR ↓Std Acc ↑
Qwen2.5-7B-InstructBEHAVE40.0694.0710.6472.44
CLPA_C 0.25×23.53▼97.2313.69▼72.44
CLPA_C 0.5×27.45▼97.2313.06▼71.33
CLPA_C 1×27.73▼96.4411.77▼72.00
CLPA_C 2×41.4693.2812.60▼71.78
Ministral-3-14B-InstructBEHAVE55.4691.7016.5772.67
CLPA_C 0.25×57.7090.5116.2771.78
CLPA_C 0.5×61.06▲91.3014.93▲71.56
CLPA_C 1×53.7894.0713.96▲71.33
CLPA_C 2×68.91▲82.21▼12.17▲71.56
Qwen3.6-27BBEHAVE98.8832.817.5582.44
CLPA_C 0.25×97.4823.72▼11.91▼84.44
CLPA_C 0.5×98.3220.95▼10.30▼85.33
CLPA_C 1×98.0429.646.6982.44
CLPA_C 2×99.1623.72▼8.0284.00

▲ and ▼ mark CLPA_C values significantly better or worse than BEHAVE on the same model. The numbers repeat Table 12 of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer ukrB (2/2): W2, Q2, and Q3

Following the suggestion of the reviewer, we incorporate CLPA_C and the corresponding ablations into the tool-use comparison. These experiments separate the design of BEHAVE from principle adaptation in general.

CLPA_C in Table 2. We include CLPA_C under the same tools, pressure dimensions, and tool naming, as the table below shows. CLPA_C is close to DPO-generic, and BEHAVE outperforms both at p<0.001. The same comparison on the six pressure dimensions of Qwen2.5-7B-Instruct leads to the same conclusion, as Table 2 of the revised paper shows.

Tool-use ablations. The table below includes two controls.

The first control is SFT, which fine-tunes on the same 37 stems without latent perturbation. SFT lowers PropensityScore relative to Base, by a much smaller margin than BEHAVE.

The second control is the frozen control of Table 4 of the revised paper, which follows the same training procedure as BEHAVE without the per-epoch refresh of the direction and the strength. Its PropensityScore under harmful tool naming rises to 45% to 52%.

The methods that adapt on the same principle data, SFT and CLPA_C, and the frozen control all obtain a partial improvement, and only the full BEHAVE reaches the lowest PropensityScore. These results suggest that the transfer to multi-turn tool use does not come from the principle data alone.

PropensityScore on Qwen3.6-27B (%)

MethodTime PP_HTime PP_BFin. PP_HFin. PP_BPower PP_HPower PP_BRes. PP_HRes. PP_BSelf PP_HSelf PP_BEval PP_HEval PP_B
Base72.789.772.489.366.285.572.389.366.586.167.085.7
★ SFT63.883.664.786.258.282.165.284.656.280.156.481.5
★ BEHAVE (frozen direction and γ_flip)51.578.351.876.047.775.052.177.550.074.145.372.3
★ CLPA_C37.172.032.571.728.766.139.474.029.267.830.966.7
DPO-generic34.771.935.572.731.867.937.373.334.269.229.068.1
BEHAVE9.746.79.345.78.443.710.848.57.442.48.544.4

★ marks the rows that are not in Table 2 of the original submission. PP_H and PP_B report PropensityScore when the misaligned tool receives a harmful name or a benign-sounding name, and lower is better. BEHAVE outperforms every other row at p<0.001 in each pressure dimension.

Dear Reviewer ukrB,

Thank you again for your constructive review. As the discussion period is closing soon, we would like to kindly check whether our responses have addressed your concerns. A brief summary of our rebuttal follows.

TL;DR

  • W1 and Q1 (paired comparison between BEHAVE and CLPA_C). We added paired exact McNemar tests with Holm correction on all metrics and on every radius of the sweep. No single CLPA_C radius matches BEHAVE on all three models, and BEHAVE sets its strength from the 37 training pairs without downstream tuning.
  • W2 and Q2 (CLPA_C in the tool-use comparison). We added CLPA_C to all six pressure dimensions on Qwen3.6-27B. Its PropensityScore under harmful tool naming is 29% to 39%, against 7% to 11% for BEHAVE (p<0.001 in each dimension). The same conclusion holds on Qwen2.5-7B-Instruct.
  • Q3 (tool-use ablations). We added SFT on the same 37 stems and the frozen control without the per-epoch refresh. Their PropensityScores are 56% to 65% and 45% to 52%, which indicates that the transfer to multi-turn tool use does not come from the principle data alone.

All new results are in the revised paper. We would greatly appreciate it if you could let us know whether any concern remains, and we are happy to provide further clarification. If our responses have resolved your concerns, we would be grateful if you would consider updating your score.

Reviewer_eZr6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether a small set of principle-derived examples can make a model prioritize harm avoidance over a conflicting task goal. The authors propose BEHAVE, which fine-tunes LoRA adapters on 37 constitution-derived contrastive stems while perturbing a hidden layer along directions that separate consistent from inconsistent continuations. The perturbation strength is calibrated to a preference-flip threshold, directions and threshold are re-estimated after each epoch. On three models, the paper reports higher harm avoidance on unseen ManagerBench scenarios with capability largely preserved, although benign-action retention depends on the base model. On one model, BEHAVE also selects the misaligned tool less often on PropensityBench, with residual sensitivity to tool naming.

Critical strengths
  • The paper addresses an important problem for agentic alignment: whether a written principle governs decisions when it conflicts with the task goal. The evaluation reports Harm Avoidance (HA) on harmful scenarios with Control Pragmatism (CP) on matched benign controls, plus capability and over-refusal benchmarks for every method, which separates selective harm avoidance from general caution. This design exposes a CP drop on Qwen3.6-27B that capability benchmarks and XSTest safe-prompt refusal miss (section 5.2). I consider this the most useful finding of the paper.
  • The paper shows that 37 principle-derived examples, none drawn from the evaluation benchmarks, change decisions in unseen operational scenarios on all three models, and the HA gain holds for every benchmarked training seed (app. D). On the two smaller models, even SFT on these examples yields a larger HA gain than DPO-targeted, which uses 5,000 task-matched scenarios (table 1). This makes compact principle-level supervision a credible alternative to task-specific preference data for specifying decision priorities.
Critical weaknesses
  • The experiments do not show that behavior-relative calibration, the main methodological contribution, improves on a well-chosen fixed perturbation strength. On Ministral-3-14B and Qwen3.6-27B, the fixed-radius baseline CLPAC\text{CLPA}_C comes within about 3 pts of BEHAVE on both HA and CP (Table 1), and the paper acknowledges that a tuned radius can approach BEHAVE. The Qwen2.5-7B advantage is confounded: only that BEHAVE run uses five continuations per stem, and only on that model are five baselines, including CLPAC\text{CLPA}_C, reported at loss-saturation checkpoints rather than after the full training budget (appendix B). A Qwen2.5-7B BEHAVE run with one continuation per stem, as in CLPAC\text{CLPA}_C, would resolve this.

  • The tool-use evidence comes from one model and does not separate harm avoidance from reduced action. PropensityBench is run only on Qwen3.6-27B (table 2), the model with the largest CP drop (Table 1). Because the aligned tool returns a persistent error, an agent that stops calling tools also scores well, and the paper does not report how often the aligned tool is attempted. A second model that retains CP, with aligned-tool attempt rates, would support the claim.

Questions for authors
  • On Qwen2.5-7B, what HA and CP does BEHAVE obtain with one continuation per stem, and did the fixed-dose controls there also use five continuations?
  • Can CLPAC\text{CLPA}_C be run with a tuned radius on all three models?
  • How does γflip\gamma_{\text{flip}} evolve over epochs, and how many refreshes reach the search ceiling without a flip?
  • How often do Base, DPO-generic and BEHAVE attempt the aligned tool on PropensityBench, and is the PPB\text{PP}_B reduction significant?
5 comments
AuthorsOpenReview ↗

Response to Reviewer eZr6 (1/4): W1 part (1) and Q2

We thank the reviewer for raising this central question and for proposing a concrete control experiment. Our reply to W1 has three parts, and this comment covers part (1).

W1 (1) and Q2. Does calibration improve on a fixed strength?

BEHAVE re-measures the strength every epoch without downstream evaluation, whereas the fixed strength that suits a model is known only after evaluation on the downstream benchmark. Three pieces of evidence support this argument.

The required strength changes during training. On Qwen2.5-7B-Instruct and Qwen3.6-27B, the preference-flip strength rises from 9.5 to 704 and from 8.75 to 960 during training, a factor of about 70 to 110.

Holding the strength at its initial value lowers HA. The fixed-dose controls in Table 13 of the revised paper fix only the strength and still refresh the direction every epoch. On all three models, the dose at k = 1, which holds the strength at its value before training, obtains significantly lower HA than BEHAVE. On Qwen3.6-27B, each of the six fixed doses obtains significantly lower HA than BEHAVE (89.36 to 96.36, against 98.88), significantly higher ASR (11.20 to 13.93, against 7.55), and significantly lower Std Acc (75.78 to 78.67, against 82.44), together with significantly higher CP (39.13 to 56.13, against 32.81), as Table 13 of the revised paper shows. The largest dose applies the late-training strength from the first epoch, and the health gate stops this run during epoch 7.

A fixed radius does not transfer across models. A tuned CLPA_C radius can reach an HA–CP balance close to BEHAVE, as Table 12 of the revised paper reports for four radii on each of the three models. The table below reports paired tests against BEHAVE for every radius, which answers Q2. The radius that matches BEHAVE on Qwen2.5-7B-Instruct significantly lowers CP on the other two models. The radius that matches BEHAVE on Qwen3.6-27B obtains significantly lower HA on Qwen2.5-7B-Instruct. On Ministral-3-14B-Instruct, the 0.5× radius outperforms BEHAVE on HA and ASR, and the same radius is significantly worse than BEHAVE on the other two models. Each of the four radii is significantly worse than BEHAVE on at least one model, and identifying the suitable radius requires evaluation on the downstream benchmark.

CLPA_C radius sweep with paired tests against BEHAVE

ModelVariantHA ↑CP ↑ASR ↓Std Acc ↑
Qwen2.5-7B-InstructBEHAVE40.0694.0710.6472.44
CLPA_C 0.25×23.53▼97.2313.69▼72.44
CLPA_C 0.5×27.45▼97.2313.06▼71.33
CLPA_C 1×27.73▼96.4411.77▼72.00
CLPA_C 2×41.4693.2812.60▼71.78
Ministral-3-14B-InstructBEHAVE55.4691.7016.5772.67
CLPA_C 0.25×57.7090.5116.2771.78
CLPA_C 0.5×61.06▲91.3014.93▲71.56
CLPA_C 1×53.7894.0713.96▲71.33
CLPA_C 2×68.91▲82.21▼12.17▲71.56
Qwen3.6-27BBEHAVE98.8832.817.5582.44
CLPA_C 0.25×97.4823.72▼11.91▼84.44
CLPA_C 0.5×98.3220.95▼10.30▼85.33
CLPA_C 1×98.0429.646.6982.44
CLPA_C 2×99.1623.72▼8.0284.00

▲ and ▼ mark CLPA_C values significantly better or worse than BEHAVE on the same model. The numbers repeat Table 12 of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer eZr6 (2/4): W1 parts (1) continued, (2), (3), and Q1

W1 (1), continued. The frozen control in multi-turn tool use

We also extend the frozen control of Table 4 of the revised paper to multi-turn tool use. This control follows the same training procedure as BEHAVE, but the direction and the strength are measured once before training and never refreshed. The PropensityScore under harmful tool naming rises from the 7% to 11% of BEHAVE to 45% to 52%, as the appendix of the revised paper reports. These results indicate that the tool-use result of BEHAVE depends on the per-epoch refresh.

W1 (2) and Q1. Does the advantage on Qwen2.5-7B-Instruct come from the five continuations?

With one continuation per stem, BEHAVE still obtains significantly higher HA than CLPA_C, with no significant difference in CP, as the table below shows. Following the suggestion of the reviewer, this run trains BEHAVE on the same stems and continuations as CLPA_C. The additional continuations further raise HA and lower ASR. The appendix of the revised paper reports this row.

For the second part of Q1, all BEHAVE variants on Qwen2.5-7B-Instruct, namely the fixed-dose controls in Figure 2, the component controls in Table 4 of the revised paper, and the seed repetitions in Appendix D, are trained on the same five-continuation data as BEHAVE. The data difference exists only between BEHAVE and the baselines, and the table below addresses this difference.

BEHAVE with one continuation per stem on Qwen2.5-7B-Instruct

MethodContinuations per stemHA ↑CP ↑
CLPA_C127.7396.44
★ BEHAVE (single continuation)132.21▲96.05
BEHAVE540.06▲94.07

▲ marks HA significantly higher than CLPA_C, at p<0.01 with one continuation and at p<0.001 with five continuations. No CP difference is significant.

W1 (3). Do the earlier baseline checkpoints on Qwen2.5-7B-Instruct favor BEHAVE?

The gap remains when the baselines are reported at epoch 30. As Appendix B states, the epoch-30 adapters of these five baselines could not be evaluated on all benchmarks, and we therefore report the checkpoints where their training losses saturate. No downstream benchmark was consulted for this choice. CLPA_C at epoch 30 obtains an HA of 24.93 and a CP of 95.26, and neither is higher than the 27.73 and 96.44 of the reported checkpoint. Among the other baselines with epoch-30 scores, the highest HA is 28.85, which remains below the 40.06 of BEHAVE.

AuthorsOpenReview ↗

Response to Reviewer eZr6 (3/4): W2 part (1)

Following the suggestion of the reviewer, we incorporate a second model and report the aligned-tool attempt rate to separate harm avoidance from reduced action.

W2 (1). A second model that retains high CP

We run all six pressure dimensions of PropensityBench on Qwen2.5-7B-Instruct, which retains high CP in Table 1. BEHAVE obtains the lowest PropensityScore in every pressure dimension under both harmful and benign tool naming, as the table below shows.

PropensityScore on Qwen2.5-7B-Instruct (%)

MethodTime PP_HTime PP_BFin. PP_HFin. PP_BPower PP_HPower PP_BRes. PP_HRes. PP_BSelf PP_HSelf PP_BEval PP_HEval PP_B
Base33.554.636.954.333.452.534.955.033.954.630.848.7
CLPA_C24.048.532.351.325.245.629.449.126.948.922.944.4
DPO-generic21.847.125.349.822.045.423.747.824.248.120.743.2
BEHAVE17.540.920.942.415.636.919.839.614.838.214.437.1

PP_H and PP_B report PropensityScore when the misaligned tool receives a harmful name or a benign-sounding name, and lower is better. BEHAVE outperforms Base and CLPA_C at p<0.001 and DPO-generic at p<0.05 in each pressure dimension, for both PP_H and PP_B.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and the Area Chair

We thank the three reviewers for their concrete suggestions.

Autonomous agents may take prohibited actions to complete a task when the sanctioned path is blocked. Our paper studies how to make a model keep a principle above the task goal when the two conflict.

BEHAVE perturbs the hidden states of the model during training toward the constitution-inconsistent continuation, and the model learns to follow the taught principle under this pressure. The key design question for this family of methods is how strong the perturbation should be. The central claim of BEHAVE is that the perturbation strength should follow the current preference of the model and be refreshed during training, instead of being a fixed geometric radius. Two results in the paper support this claim, and the analyses suggested by the reviewers extend them.

1. BEHAVE sets the perturbation strength without downstream tuning.

BEHAVE re-measures the preference-flip strength every epoch from the 37 training pairs, and no downstream benchmark enters this step.

The preference-flip strength rises from 9.5 to 704 on Qwen2.5-7B-Instruct and from 8.75 to 960 on Qwen3.6-27B during training, as the appendix of the revised paper plots for every epoch. The fixed-dose controls in the paper keep the strength at a multiple k of the value measured before training and still refresh the direction. On all three models, the dose at k = 1 obtains significantly lower Harm Avoidance (HA) than BEHAVE, as Table 13 of the revised paper shows. On Qwen3.6-27B, every fixed dose obtains significantly worse HA, attack success rate (ASR), and uninjected accuracy (Std Acc) than BEHAVE and significantly higher Control Pragmatism (CP), as the same table shows.

CLPA_C is a latent adversarial training method with the same 37 stems as BEHAVE and a fixed perturbation radius. A tuned radius can approach the HA and CP of BEHAVE on individual models, as Section 6.1 of the paper reports. The radius sweep in the appendix of the revised paper shows that no single radius matches BEHAVE on all three models, and the radius that suits a model is known only after evaluation on the downstream benchmark.

2. The taught priority holds under pressure and transfers to tool use that the training data never cover.

The training data of BEHAVE contain only 37 first-person stems with short continuations, with no tool call and no multi-turn interaction.

PropensityBench gives the agent an aligned tool and a misaligned tool that completes the same task through a prohibited action. The aligned tool keeps returning an error while one of six operational pressures, such as a deadline or a financial loss, rises over later turns. PropensityScore is the percentage of scenarios in which the model invokes the misaligned tool.

The model trained with BEHAVE still attempts the aligned tool in the large majority of scenarios. On Qwen3.6-27B under harmful tool naming, Base invokes the misaligned tool in 66% to 73% of the scenarios, while BEHAVE does so in 7% to 11%, as Table 2 of the paper shows. After the aligned tool fails on this model, BEHAVE gives up the task instead of violating the principle, as Section 5.3 of the revised paper reports. On two models and six pressure dimensions, BEHAVE obtains the lowest PropensityScore among the compared methods, as Table 2 of the revised paper shows. The frozen control in the paper follows the same training procedure as BEHAVE without the per-epoch refresh of the direction and the strength, and its PropensityScore under harmful tool naming rises to 45% to 52%.

Consistency of the experimental setup. The conclusion holds under a matched setup. With one continuation per stem, which gives the same stems and continuations as CLPA_C, BEHAVE still obtains significantly higher HA than CLPA_C, as the appendix of the revised paper reports.

Table numbers. Table numbers in our responses follow the revised paper. The revised paper adds Table 3 in Section 5.3, and Tables 3 to 12 of the original submission become Tables 4 to 13.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.