acceptodds
Under review as a conference paper at ICLR 2027

Self-Anchored Reference Distributions for Multi-Objective Foundation Model Alignment

Abstract

Foundation model (FM) alignment is inherently a multi-objective optimization problem that requires balancing competing goals, such as suppressing undesirable behaviors while complying with legitimate requests. Existing solutions typically optimize weighted combinations of objective-specific losses or reconcile conflicting gradients, but often require extensive hyperparameter tuning and may favor certain sub-objectives. We identify a central need for effective multi-objective alignment: characterizing an ideal posterior distribution of each alignment objective that provides richer supervision while reducing negative transfer. Motivated by FMs' few-shot learning ability to produce steerable predictive distributions, we propose a divide and conquer strategy that constructs objective-specific supervision from the target model itself. Specifically, the model is conditioned on goal-specific instructions, which generates self-supervised teacher distribution that better characterizes the target distribution yet mitigates the distributional divergence across alignment goals. We evaluate our method in two settings where goal conflicts are especially pronounced: unlearning and safety alignment. Comprehensive evaluations show that our method achieves the strongest overall balance among competing objectives without sacrificing performance on individual learning objectives.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Foundation Model Alignment, Multi-Objective Optimization, Cross-Objective Interference, Self-Distillation, Machine Unlearning, Safety Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_UQ3DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper frames foundation-model alignment as multi-objective optimization and argues that cross-objective interference stems partly from the supervision targets themselves, not only from how losses are aggregated. The proposed method, CAESAR, works as follows:

  • For each objective oo, the frozen base model is conditioned on an objective-specific instruction coc_o and the gold response yy as privileged context.
  • The resulting distribution π^o(⋅∣co,x,y<t)\hat\pi_o(\cdot\mid c_o,x,y_{<t}) serves as a soft teacher.
  • This teacher is distilled into the student along gold-response prefixes using a bidirectional top-KK Smooth-L1 loss on raw logits, applied over both teacher-selected and student-selected supports (Eq. 5).
  • Each minibatch is drawn from a single objective, so no loss weights λo\lambda_o are tuned.

A pilot study on WildJailbreak shows that one-batch SFT updates create cross-objective NLL interference (Fig. 1), which is reduced when the SFT target is replaced by the self-anchored teacher (Fig. 2, Fig. 5).

Experiments cover two unlearning benchmarks and one safety benchmark:

  • MUSE-Book (Llama-3.2-3B) and WMDP-Cyber (Zephyr-7B), each with four objectives: forgetting, general retention, neighbor retention and prefilling robustness.
  • WildJailbreak safety alignment (Tulu-2-7B), with direct/adversarial harmful and benign requests.

With 10 seeds and matched tuning budgets, CAESAR obtains the best aggregate "Overall" score in all three settings (Tables 1–3), with margins of +36.8, +44.9 and +31.5 points over the strongest baseline. Ablations cover reference conditioning, on-policy prefixes, alternative distillation losses, top-KK, and limited or imbalanced data (Tables 4–7).

Critical strengths

1. A simple, practical recipe with consistently strong results. CAESAR needs no reward model, no preference pairs, no separate teacher and no per-objective loss weighting. The method is easy to adopt, and the evaluation is reasonably careful: 10 seeds, a matched hyperparameter-search budget across methods, and validation-based checkpoint selection (Section 5.1). Gains appear across two unlearning benchmarks and one safety benchmark on three base models. The low-data and imbalanced-data results are especially notable. For example, under benign scarcity, adversarial benign refusal is 23.4% for CAESAR vs. 78.1% for SFT and 70.3% for Lisa, at equal adversarial ASR to Lisa (Table 7). These results suggest that soft self-generated targets are genuinely more data-efficient than one-hot demonstrations for refusal/compliance trade-offs.

2. Supervision construction is a useful and under-explored lens on interference. Most multi-objective alignment work (PCGrad, NGDiff, MOLLM) acts on gradients or weights after the targets are fixed. The paper instead asks how each objective's target distribution affects the others, and the per-cell interference matrix (Figs. 1–2) is an informative diagnostic. Fig. 4 is a concrete illustration of why soft targets help: "assist" and "help" both yield refusals, but SFT rewards only one of them.

Critical weaknesses

1. The interference analysis is confounded by update magnitude, and the method's novelty over context/self-distillation is limited.

  • Smaller interference may just reflect smaller updates. The self-anchored teacher is the base model itself, so distilling toward it moves the student much less than an SFT step toward one-hot targets. The diagonal of the interference matrix confirms this. Within-objective NLL gains are also smaller under CAESAR for three of four objectives: Δ1,1\Delta_{1,1} −0.637 vs. −1.479, Δ2,2\Delta_{2,2} −1.425 vs. −2.076, and Δ3,3\Delta_{3,3} −0.878 vs. −1.289 (Fig. 2 vs. Fig. 1). The reduced off-diagonal interference may therefore reflect a smaller effective step rather than better cross-objective compatibility.
  • The divergence measurement is circular. Fig. 5 measures KL to a fixed objective-jj teacher, which is the same base model under another prompt. A student distilled from that model family will trivially stay closer to it.

A fair diagnostic would compare students at matched within-objective gain or matched divergence from θ0\theta_0. Without it, the central claim that self-anchored supervision is "the dominant factor in reducing cross-objective interference" (Contribution 1) is not established.

On novelty, the method combines context distillation (Askell et al., 2021; Snell et al., 2022) with privileged-information self-distillation of the OPSD/SDFT type. It is closest to DUET (Zhong et al., 2026), which already uses prompt-steered self-supervision for unlearning. The incremental elements are per-objective prompts, reference conditioning and the Smooth-L1 logit loss. Related work in which models learn from spec-conditioned versions of themselves, such as deliberative alignment (Guan et al., 2024), should also be discussed.

2. The comparison likely favors CAESAR through prompt engineering and an aggregate metric that does not match the paper's own motivation.

  • Teacher instructions encode benchmark-specific knowledge. The MUSE prompts (Appendix 8.1) are long, hand-tuned and benchmark-aware. For example, the prefilling prompt enumerates the exact neighbor works to keep answering (The Tales of Beedle the Bard, Fantastic Beasts, The Lord of the Rings). This injects knowledge of the evaluation's neighbor and forget boundaries into the supervision, which baselines trained on the same examples do not receive. The paper does not say how these instructions were developed or whether test data was consulted. Notably, "w/o ref. answer" already beats every baseline by a wide margin, which suggests the instructions carry much of the gain.
  • Several baselines look degraded. GA increases Forget on MUSE-Book (36.9 vs. 32.1 for the base model; Table 1). SafeDecoding, SmoothLLM and Global Teacher drop MMLU to 28.0/27.5/34.4 and GSM8K to as low as 10.0 (Table 2), and Table 2 has no base Tulu-2-7B row against which to judge these drops.
  • The aggregate rewards large swings, not balance. "Overall" is an unweighted sum of raw changes in heterogeneous metrics (ROUGE, accuracy, ASR, MMLU), so large swings on one axis dominate; on MUSE-Book, CAESAR's ASR drop alone contributes about 58 points. The introduction argues that reliable alignment "cannot tolerate failure on any single objective", which calls for a worst-objective or normalized metric rather than a sum.

The headline margins of 31–45 points therefore overstate the evidence, and it is unclear how much of the improvement survives with generic instructions and properly tuned baselines.

Questions for authors
  1. Can you report the Fig. 1/2 interference comparison at matched within-objective improvement, or at matched KL from θ0\theta_0? For example, scale the SFT learning rate so that Δi,i\Delta_{i,i} equals CAESAR's, and then compare the off-diagonal cells.
  2. How were the teacher instructions in Appendix 8 developed: on training data, validation data, or after inspecting test results? Do the works named in the MUSE prefilling prompt overlap with the neighbor or forget evaluation sets? How does CAESAR perform with a short, generic instruction per objective that names no benchmark entities?
  3. Please add the base Tulu-2-7B row to Table 2. Why do SafeDecoding, SmoothLLM and Global Teacher lose up to 30 MMLU points? Why does GA fail to reduce Forget on MUSE-Book? Were these baselines tuned with the same validation budget?
  4. Do the method rankings in Tables 1–3 hold under a normalized or worst-objective aggregate (e.g., the minimum per-objective improvement relative to the base model), which matches the stated motivation better than the unweighted sum?
  5. Beyond per-objective prompts and the Smooth-L1 loss, what distinguishes CAESAR from DUET and from standard context distillation? In particular, does DUET with your per-objective prompts close most of the gap?
6 comments
AuthorsOpenReview ↗

Response to W1 (Part 1)

W1 (confound of update magnitude)

Thank you for the thoughtful comments. In Figures 1 and 2, each method used its own training configuration, and the update size was indeed not controlled. Following the suggestion, we redid the comparison under matched conditions: both methods are read out at the same displacement from the initial parameters (∥θ−θ0∥=1.35\lVert\theta-\theta_0\rVert=1.35, the minimum of the final displacements over all runs, chosen without looking at NLL; 3 seeds). The metric is the same as in Figure 1. It is the change in NLL on held-out reference answers of each objective, Δ(i→j)\Delta(i\to j) (nats/token, lower is better), and it does not depend on any teacher.

The full results are given as heatmaps in the updated Figure 1 (SFT) and Figure 2 (CAESAR) of the revised manuscript.

(1) At matched update size, the cross-objective interference of CAESAR is still lower. Among the 8 objective pairs with opposite decisions (one requires refusal and the other requires answering), CAESAR is lower on 7, and the average drops from +0.132 to +0.040 nats/token. The average over all 12 off-diagonal cells is +0.005, compared with +0.079 for SFT.

(2) The improvement on the objective itself is not smaller. At matched update size, the diagonal average is -0.187, compared with -0.117 for SFT. CAESAR improves more on Benign and Adv. benign (-0.181 vs. -0.008, and -0.248 vs. -0.054), and slightly less on Harmful and Adv. harmful (-0.063 vs. -0.083, and -0.256 vs. -0.321).

(3) At the behavior level, comparing at similar objective performance gives the same conclusion. When direct harmful ASR is around 3% to 4%, the direct / adversarial benign refusal rates of CAESAR are 7.19% / 4.59%, compared with 18.74% / 14.87% for SFT and 14.26% / 41.68% for Lisa.

We have also changed "the dominant factor" in Contribution 1 to "a crucial factor", so that the statement matches the evidence.

W1 (divergence measure in Figure 5)

Thank you for pointing this out. We would like to clarify that in Figure 5 both students start from the same θ0\theta_0 and are compared with the same fixed reference under the same questions and prefixes, and the CAESAR student was not trained with the teacher of objective jj. We agree that this measure uses the self teacher as the reference. The main evidence of this paper is the NLL interference matrix, which does not depend on any teacher (Figures 1 and 2), and the two lead to the same conclusion.

AuthorsOpenReview ↗

Response to W1 (Part 2)

W1 (novelty)

Thank you for pointing out these related works. We have added a discussion of Askell et al. (2021), SDFT, and deliberative alignment (Guan et al., 2024) to the Related Work of the revised manuscript. We would like to highlight the following two points.

(1) One of our contributions is identifying an important yet overlooked problem (pages 1-2, lines 48-55). Some of the prior work above shares common ground with us in using contextualized distillation or self-distillation, but it focuses on task-specific performance. The key difference lies in the problem we study: what causes cross-objective interference in multi-objective alignment, and how to construct supervision to mitigate it? This question leads to findings that are not covered by prior work on contextualized distillation:

  • Interference originates partly from the supervision target itself. Under controlled diagnostics at matched update magnitude, one-hot supervision induces substantially more interference on opposite-decision objectives than self-anchored supervision (+0.132 vs. +0.040 nats/token; Section 3.2 and Figures 1-2, pages 3-4), and our analysis traces this to hard labels suppressing valid alternative phrasings (Section 5.4, pages 9-10, and Appendix 7.9).
  • How the teacher is constructed determines the interference between conflicting objectives. A single teacher that encodes all requirements, as in DUET, gives conflicting supervision across objectives. In a matched control that changes only this choice, merging the per-objective teachers into one reduces Overall on WildJailbreak from 64.65 to 30.64 and raises adversarial ASR from 2.00% to 25.34% (Section 5.4, page 10, and Table 11 in Appendix 7.10).
  • Anchoring the teacher to the student's own base model matters. Our additional experiments show that, replacing the self-anchored teacher with an external model of the same size or larger reduces Overall to between 4.92 and 30.83 (Section 5.4, page 10, and Table 12 in Appendix 7.11).

(2) Regarding Q5: giving DUET our per-objective teachers does close most of the gap, which is exactly bridged by this paper. On MUSE-Book, the Overall of DUET is 5.52. Keeping the DUET loss and only replacing the supervision with per-objective, reference-conditioned self-anchored teachers gives 74.33, and the full CAESAR gives 87.32 (Table 1 and Table 4). About 84% of the gap is closed by the construction of the supervision, and the rest comes from the bidirectional loss. Each part of the supervision construction is also isolated separately (WildJailbreak, where the full CAESAR is at 64.65). Replacing the per-objective instructions with one shared instruction gives an Overall of 30.64. Replacing the self teacher with an external teacher gives between 4.92 and 30.83. Removing the reference answer gives 46.76 (Table 11 in Appendix 7.10).

AuthorsOpenReview ↗

Response to W2 (Part 1)

W2 (teacher instructions)

Thank you for the valuable comments.

(1) How the instructions were written. All instructions were written only from the task definitions, the training set, and the validation set, without consulting test data or test results. The works named in the MUSE prefilling instruction are the works covered by the neighbor-retention training data. All baselines receive the same forget / retain boundary in the form of training examples, and the instruction only states this boundary in natural language. The neighbor evaluation set targets the same works, and its questions do not overlap with the training questions. This explanation has been added to Appendix 8 of the revised manuscript.

(2) Short generic instructions. Following this valuable suggestion, on WildJailbreak we replaced the instruction of each objective with a short generic instruction that names no benchmark entity. The full results and the instruction texts are in Table 11 and Appendix 7.10 of the revised manuscript.

With short generic instructions, Overall is 39.11, which is still higher than all baselines in Table 2 (the highest is 33.17 for SFT), and MMLU and GSM8K stay unchanged. So the advantage of CAESAR does not depend on instructions written for the benchmark. The more specific the instruction is, the more accurate the supervision given by the teacher is, which is consistent with the claim of this paper that the quality of the supervision signal is the key. In addition, the original instructions for WMDP-Cyber and WildJailbreak already name no benchmark entity (Appendix 8.2 and 8.3).

(3) On "w/o ref. answer already beats every baseline". This is consistent with the claim of this paper: supervision constructed per objective plays a crucial role. But the gain does not come from the information contained in the instructions themselves. When all requirements are written into one instruction and the model is prompted directly (Global Teacher), Overall is 28.52 on MUSE-Book and -13.11 on WildJailbreak. Distilling with such a shared instruction also gives only 30.64 (Table 11). The gain comes from constructing the supervision separately for each objective.

W2 (baselines)

Thank you for the suggestion. The base Tulu-2-7B row has been added to Table 2 of the revised manuscript. Relative to the base model, CAESAR lowers direct / adversarial harmful ASR from 21.88% / 60.57% to 3.69% / 2.00%, the benign refusal rates are almost unchanged, and MMLU drops by 2.3 points.

(1) The MMLU drop of SafeDecoding, SmoothLLM, and Global Teacher. This comes from the conflict between the safety objectives and general capability. The safety interventions of these three methods (the predictions of a safety expert, input perturbation, and a goal-prioritization prompt) do not distinguish which objective a request belongs to, so they harm the general capability of the model while lowering ASR. This conflict is exactly the problem this paper addresses: CAESAR constructs the supervision separately for each objective, and its MMLU is only 2.3 points lower than that of the base model.

(2) GA. All methods use the same criterion to select checkpoints: the highest Overall on the validation set (Section 5.1). GA only maximizes the loss on the forget data, its training is unstable, and the degradation of general capability appears before forgetting. At the selected checkpoint, MMLU has already dropped from 60.8 to 36.45, while Forget has not decreased yet (36.9, compared with 32.1 for the base model). Under the same criterion, a checkpoint with lower Forget would not have a higher Overall, so it was not selected. A similar trade-off can also be seen in Figure 8 (Appendix 7.8) of the revised manuscript (WildJailbreak): when a baseline pushes the ASR on harmful requests lower, its benign refusal rate goes up.

(3) Tuning budget. All baselines use the same hyperparameter search budget and checkpoint selection criterion, and the method-specific weights are also within the budget (Section 5.1, lines 365-367).

Reviewer_6So6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper studies multi-objective alignment of LLMs. The authors argue that this interference of multi-objectives stems not only from loss aggregation or gradient conflicts but also from the supervision target itself: SFT's one-hot targets on limited demonstrations induce negative transfer across objectives. To solve this problem, they propose Caesar: For each objective, a "self-anchored" teacher is built by conditioning the frozen base model on an objective-specific instruction and on the sample's gold reference response. This teacher is distilled back into the instruction-free student on gold-answer prefixes. Experiments cover unlearning on MUSE-Book, WMDP-Cyber, and safety alignment on WildJailbreak.

Critical strengths
  1. Caesar provides a simple and direct way to solve the multi-objectives interference problem.
  2. Overall improves from 50.52 to 87.32 on MUSE-Book (Table 1), from 71.70 to 116.60 on WMDP-Cyber (Table 3), and from 33.17 to 64.65 on WildJailbreak (Table 2). On safety, Caesar lowers adversarial ASR and adversarial benign refusal at the same time (2.00% / 4.59% vs. 7.35% / 14.80% for SFT), and the distilled student even exceeds the Oracle-routed Teacher on safety (64.65 vs. 58.59).
Critical weaknesses
  1. My main concern is limited novelty. The core idea, conditioning the model on the gold response so that it rewrites the target in its own distribution, is the idea of SDFT [R1] (which is not cited). The method is also close to context distillation [R2], DUET [R3], and self-distillation with privileged information [R4, R5]. Compared with DUET [R3], the main differences are per-objective instructions, conditioning on the reference answer, and a student-side top-KK term.
  2. Caesar combines several changes: a self-anchored teacher, soft labels, reference conditioning, and per-objective instructions, and its updates also stay close to the base model. There is no ablation with an external teacher. There are also no conservative SFT baselines (e.g., SFT with a KL-to-base penalty, LoRA), which would show whether the gains simply come from smaller updates.
  3. The safety evaluation is largely in-distribution: training and adversarial test data both come from WildJailbreak, and other attack families and over-refusal benchmarks (OR-Bench) are missing.
  • [R1] Z. Yang, T. Pang, H. Feng, H. Wang, W. Chen, M. Zhu, Q. Liu. Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning. ACL 2024.
  • [R2] C. Snell, D. Klein, R. Zhong. Learning by Distilling Context. https://arxiv.org/abs/2209.15189
  • [R3] Y. Zhong, Z. Yang, Z. Zhu. DUET: Distilled LLM Unlearning from an Efficiently Contextualized Teacher. ICLR 2026.
  • [R4] J. Hübotter et al. Reinforcement Learning via Self-Distillation. https://arxiv.org/abs/2601.20802
  • [R5] Y. Ma, Z. Zhu, M. Jiang, C. Xiao. One Student, Many Teachers: Multi-Task On-Policy Distillation via Soft-Prompt Privileged Context. https://arxiv.org/abs/2607.18293
Questions for authors
  1. Can you compare performance when using a larger teacher model, given the same instruction and reference?
  2. Is the forgotten knowledge actually removed? The teacher prompt for forget queries asks for a fixed refusal, and Forget is measured by ROUGE, so a model that only learns to refuse would already score well.
  3. Why does GA's Forget score exceed the base model's, and which validation criterion was used for checkpoint selection?
5 comments
AuthorsOpenReview ↗

Response to W1

W1 (novelty; SDFT)

Thank you for the valuable comments, and thank you for pointing out SDFT [R1]. We have cited and discussed it in the revised manuscript. [R2] to [R5] are already discussed in the Related Work (lines 300-303 of the revised manuscript). We would like to highlight the following three points.

(1) One of our contributions is identifying an important yet overlooked problem (pages 1-2, lines 48-55). Some of the prior work above shares common ground with us in using contextualized distillation or self-distillation, but it focuses on task-specific performance. The key difference lies in the problem we study: what causes cross-objective interference in multi-objective alignment, and how to construct supervision to mitigate it? This question leads to findings that are not covered by prior work on contextualized distillation:

  • Interference originates partly from the supervision target itself. Under controlled diagnostics at matched update magnitude, one-hot supervision induces substantially more interference on opposite-decision objectives than self-anchored supervision (+0.132 vs. +0.040 nats/token; Section 3.2 and Figures 1-2, pages 3-4), and our analysis traces this to hard labels suppressing valid alternative phrasings (Section 5.4, pages 9-10, and Appendix 7.9).
  • How the teacher is constructed determines the interference between conflicting objectives. A single teacher that encodes all requirements, as in DUET, gives conflicting supervision across objectives. In a matched control that changes only this choice, merging the per-objective teachers into one reduces Overall on WildJailbreak from 64.65 to 30.64 and raises adversarial ASR from 2.00% to 25.34% (Section 5.4, page 10, and Table 11 in Appendix 7.10).
  • Anchoring the teacher to the student's own base model matters. Our additional experiments show that, replacing the self-anchored teacher with an external model of the same size or larger reduces Overall to between 4.92 and 30.83 (Section 5.4, page 10, and Table 12 in Appendix 7.11).

(2) The difference from SDFT is substantive. SDFT lets the model rewrite the reference answer in its own words and then runs hard-label fine-tuning on the rewritten text, in order to reduce the distribution shift in single-task fine-tuning. CAESAR does not generate rewritten text. At each prefix position, it directly uses the token distribution of the model conditioned on "objective instruction + reference answer" as soft supervision, and it builds a separate teacher for each objective. This difference matters for multi-objective interference: the new analysis shows that hard labels suppress other correct phrasings (SFT lowers their log-prob by 0.80, while CAESAR raises it by 0.65). When only the top-1 token of the teacher distribution is kept (K = 1), Overall drops to 31.19, comparable to 33.17 for SFT.

(3) Each difference from DUET has a quantified effect. On MUSE-Book, the Overall of DUET is 5.52. Keeping the DUET loss and only replacing the supervision with per-objective, reference-conditioned self-anchored teachers gives 74.33. Adding the student-side Top-K term then gives 87.32 (Table 1 and Table 4).

AuthorsOpenReview ↗

Response to W2 and W3

W2 (decomposition of the changes; external teachers; conservative SFT)

Thank you for the valuable suggestion. We added an external-teacher ablation and a conservative SFT baseline.

(1) External-teacher ablation. The student is fixed to Tulu-2-7B. The instructions, reference answers, and training budget are unchanged, and only the teacher is replaced. For each external teacher we report two rows. One uses the CAESAR loss directly. The other uses a re-anchored target that keeps only the task correction, z~=z0+(zE,i−zE,0)\tilde z=z_0+(z_{E,i}-z_{E,0}), where z0z_0 is the initial logit of the student, and zE,iz_{E,i} and zE,0z_{E,0} are the logits of the external teacher with and without the instruction and reference answer. The full results are in Table 12 (Appendix 7.11) of the revised manuscript. With the CAESAR loss, Overall is 5.83 with Llama-2-7b-chat (same size, different family), 30.83 with Tulu-2-13B (same family, larger), and 4.92 with Llama-2-13b-chat (different family, larger), compared with 64.65 for the self teacher. With the re-anchored target, Overall is 13.00, 33.64, and 0.46, respectively.

The self teacher achieves the best performance, while a larger external teacher does not bring more benefits. When an external teacher is used directly, the student shows over-refusal (the benign refusal rates with the Llama teachers are 28% to 38%) or a drop in utility (GSM8K is 5.5 with Tulu-2-13B). Our explanation is that the supervision from an external teacher contains two parts: the task correction brought by the instruction and the reference answer, and the logit difference that already exists between the two models. The student is forced to fit the latter as well. After the latter is removed (re-anchored), the benign refusal rates return to between 6% and 10% and GSM8K returns to between 30.5 and 33.5, which supports this explanation. However, adversarial harmful ASR goes back up to between 33% and 46%, which shows that the task correction of an external teacher cannot be transferred directly to the logits of the student. Only the self teacher satisfies both conditions: there is no difference between models, and the task correction is on the same logit scale as the student.

(2) Conservative SFT. We added SFT + KL-to-base (β∈{0.1,1,10}\beta\in\lbrace 0.1, 1, 10\rbrace, 3 seeds each, averaged; Table 13 in Appendix 7.12 of the revised manuscript). The KL penalty raises the Overall of SFT from 33.17 to 46.53, but it is still 18.1 lower than CAESAR. The gap comes from adversarial harmful ASR (10.04% vs. 2.00%) and the benign refusal rates (direct 12.70% vs. 7.19%, adversarial 10.80% vs. 4.59%). In addition, when the two methods are compared at the same displacement from the initial parameters (∥θ−θ0∥=1.35\lVert\theta-\theta_0\rVert=1.35), CAESAR has lower cross-objective interference on 7 of the 8 objective pairs with opposite decisions (+0.040 vs. +0.132 nats/token on average). The gain does not come from smaller updates.

(3) Individual effect of each change. We change only one item at a time, and the other settings are the same as in CAESAR (WildJailbreak, where the full CAESAR is at 64.65; Table 13 in Appendix 7.12 of the revised manuscript). Removing reference-answer conditioning gives an Overall of 46.76. Replacing the per-objective instructions with one shared instruction gives 30.64. Replacing the self teacher with an external teacher gives between 4.92 and 30.83. Keeping only the top-1 token of the teacher's soft distribution (K = 1) gives 31.19. On MUSE-Book, replacing the loss with the DUET or BiLD loss lowers Overall from 87.32 to 74.33 and 69.72 (Table 4).

All of the ablations above have been added to the revised manuscript.

W3 (out-of-distribution safety evaluation)

Thank you for the valuable suggestion. We added two evaluations outside the training distribution. All models use their existing weights without retraining. (i) The 100 harmful behaviors of JailbreakBench combined with 5 types of public attacks (GCG, direct request, JBC-AIM, PAIR, random-search), with the same set of attack prompts for all methods. (ii) OR-Bench, where the hard subset (200 prompts) looks sensitive but should be answered, and the toxic subset (100 prompts) is truly harmful. The full results are in Table 14 (Appendix 7.13) of the revised manuscript.

The ASR of CAESAR is at most 3.7% on all five types of attacks, while the base model is at 91% and 96% on JBC-AIM and random-search. At the same time, the OR-Bench-hard refusal rate of CAESAR is 26.2%, which is the lowest among all methods and also lower than 38.3% for the base model. The ASR of SFT is close to that of CAESAR, but its OR-Bench-hard refusal rate is 50.8%, about twice that of CAESAR. Each of the other baselines is clearly worse on at least one item.

AuthorsOpenReview ↗

Response to Q1, Q2, and Q3

Q1

Thank you for your valuable question. A larger teacher is not better. With the same instructions, reference answers, and budget, Overall is 30.83 when the teacher is replaced by Tulu-2-13B and 4.92 when it is replaced by Llama-2-13b-chat. Both are lower than 64.65 with the self teacher (see Table 12 in Appendix 7.11 of the revised manuscript for the full results, and point (1) of our response to W2 for the explanation).

Q2

Thank you for the constructive question. We would like to highlight the following three points.

(1) The meaning of forgetting. We agree that low likelihood or ROUGE alone does not demonstrate complete removal of target knowledge from the model parameters. Our goal herein is consistent with LLM approximate unlearning research (Eldan & Russinovich, 2023; Maini et al., 2024; Shi et al., 2025), which alters the model's output distribution and suppresses the likelihood of responses associated with the forget data. This operational notion of empirical forgetting is widely adopted in current approximate LLM-unlearning research. We deem provable and exact unlearning an important research direction, but it is out of the scope of this paper, which focuses on improving multi-objective alignment.

(2) The evaluation does not rely on ROUGE alone. For prefilling ASR, an LLM judge decides whether a response reveals the target knowledge. We also added an LLM-judge-based Forget, where the judge decides whether a response gives the reference answer or an equivalent statement (Table 15 in Appendix 7.14 of the revised manuscript). The Forget of CAESAR is 2.3 with ROUGE and 3.2 with the LLM judge, compared with 32.1 and 34.3 for the base model. The two measures lead to the same conclusion, and the ranking of the non-oracle methods is identical.

(3) A model that only refuses cannot pass the retention objectives. The General of CAESAR stays at 84.3 (84.3 for the base model) and its Neighbor is 58.5 (57.28 for the base model), which shows that it does not refuse all questions about neighboring domains and general knowledge. SFT, which over-suppresses, lowers General to 49.6.

Q3

All methods use the same criterion to select checkpoints: the highest Overall on the validation set (Section 5.1). GA only maximizes the loss on the forget data, its training is unstable, and the degradation of general capability appears before forgetting. At the selected checkpoint, MMLU has already dropped from 60.8 to 36.45, while Forget has not decreased yet (36.9, compared with 32.1 for the base model). Under the same criterion, a checkpoint with lower Forget would not have a higher Overall, so it was not selected. A similar trade-off can also be seen in Figure 8 (Appendix 7.8) of the revised manuscript (WildJailbreak): when a baseline pushes the ASR on harmful requests lower, its benign refusal rate goes up.

Thank you again for your valuable suggestions, which have helped strengthen our paper. We look forward to your further feedback.

Reviewer_yfbyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces Caesar, a multi-objective alignment method that distills from a frozen copy of the initial model conditioned on objective-specific instructions and complete reference answers. It transfers this supervision to the student through bidirectional top-K logit matching. Extensive experiments on unlearning and safety evaluate its ability to balance target behaviors with retained utility.

Critical strengths
  1. Caesar uses a frozen copy of the initial model as its teacher and requires neither separately trained teachers nor privileged information at inference.

  2. The experiments examine both target behavior and retained utility across unlearning and safety settings. Evaluations with limited or imbalanced data further illustrate the method’s potential usefulness.

  3. The ablations on reference answers, prefixes, and the loss function help interpret the results. Matched tuning budgets and evaluation across ten seeds also strengthen the empirical comparisons.

Critical weaknesses
  1. A matched distillation control using a shared teacher instruction, with the same reference answers, prefixes, loss, and training budget, would help isolate the contribution of objective-specific conditioning to the reduction in interference. Figures 1–2 also show smaller within-objective NLL improvements for Caesar on three of the four objectives, leaving differences in effective update strength as a possible confound. Comparing interference and behavioral outcomes at similar levels of within-objective progress would help clarify the role of these differences in the reported results.

  2. The K sweep reports only the aggregate Overall score, while the main results across ten seeds are presented without measures of variability. On MUSE-Book, SFT achieves stronger forgetting and prefilling resistance, whereas Caesar preserves more utility. Per-objective sensitivity curves with uncertainty estimates, together with comparisons at matched forgetting or attack-success levels, would help assess whether the improved trade-offs hold beyond the selected configurations.

  3. Refusal and prefilling resistance are explicitly trained, while target forgetting is assessed mainly through open-ended answer overlap and prefilling ASR. A complementary evaluation of target knowledge that is less affected by refusal behavior, or testing on an attack family held out from training, would help clarify the extent to which the results reflect knowledge removal rather than learned refusal. Without such evidence, it would be more appropriate to frame the findings as reduced disclosure under the evaluated conditions.

Questions for authors

I would appreciate the authors' clarification of the points raised in the weaknesses above.

3 comments
AuthorsOpenReview ↗

Response to W1

W1 (matched control with a shared instruction; effective update strength)

Thank you for the valuable suggestions! We added both sets of experiments. Results show that our objective-specific conditioning is essential: replacing it with a shared instruction, with everything else fixed, halves Overall (64.65 → 30.64). The reduction in interference persists at matched update size, where CAESAR also makes larger average within-objective progress, and at matched harmful ASR, CAESAR achieves lower benign refusal.

(1) Matched distillation control with a shared teacher instruction. The reference answers, prefixes, loss, and training budget are exactly the same as in CAESAR. Only the per-objective instructions are replaced by one shared instruction that contains all requirements. The full results are in Table 11 (Appendix 7.10) of the revised manuscript.

Utility is almost unchanged, but adversarial harmful ASR rises from 2.00% to 25.34%, and the direct benign refusal rate rises from 7.19% to 13.33%. Our explanation is as follows. The shared instruction describes both the "refuse" and the "answer" requirements, so the condition of the teacher is no longer specific to the current objective, and the supervision it gives conflicts with itself between the two types of requirements. Per-objective conditioning avoids this conflict. This control has been added to Section 5.4 (page 10) of the revised manuscript.

(2) Effective update strength. In Figures 1 and 2 of the original paper, each method used its own configuration, and the update size was not controlled. We read out both methods at the same displacement from the initial parameters (∥θ−θ0∥=1.35\lVert\theta-\theta_0\rVert=1.35, 3 seeds). The metric is the same as in Figure 1, i.e., Δ(i→j)\Delta(i\to j) (nats/token, lower is better).

The full results are given as heatmaps in the updated Figure 1 (SFT) and Figure 2 (CAESAR) of the revised manuscript.

Among the 8 objective pairs with opposite decisions (one requires refusal and the other requires answering), CAESAR has lower interference on 7, and the average drops from +0.132 to +0.040 nats/token. At matched update size, the improvement on the objective itself is not smaller: the diagonal average is -0.187, compared with -0.117 for SFT. CAESAR improves more on Benign and Adv. benign (-0.181 vs. -0.008, and -0.248 vs. -0.054), and slightly less on Harmful and Adv. harmful (-0.063 vs. -0.083, and -0.256 vs. -0.321).

(3) At the behavior level, comparison at similar objective performance. On WildJailbreak, CAESAR, SFT, GKD, and Lisa are each swept over a set of update budgets, which gives the safety and compliance trade-off curves. See Figure 8 (Appendix 7.8) of the revised manuscript. At similar ASR on harmful requests, the benign refusal rate of CAESAR is clearly lower. When direct harmful ASR is around 3% to 4%, the direct / adversarial benign refusal rates of CAESAR are 7.19% / 4.59%, compared with 18.74% / 14.87% for SFT and 14.26% / 41.68% for Lisa. When adversarial harmful ASR is around 7% to 10%, CAESAR is at 5.36% / 3.94% and SFT is at 18.74% / 14.87%.

With adversarial harmful ASR on the x-axis, all 12 baseline configurations are dominated by some CAESAR configuration (neither metric is better), and no CAESAR configuration is dominated by a baseline.

AuthorsOpenReview ↗

Response to W2 and W3

W2 (per-objective results for K; variance; comparison at matched levels)

Thank you for the valuable suggestions. Our additional experiments show that CAESAR's advantage is robust: it holds per objective across K, exceeds seed-to-seed variability by a wide margin, and persists along the full trade-off curves, not only at the selected configuration.

(1) Per-objective results for K. Figure 7 (Appendix 7.4) of the revised manuscript now shows per-objective curves, which give each metric at the eight values of K. K mainly affects the refusal of harmful requests: adversarial harmful ASR drops from 26.34% at K = 1 to 2.00% at K = 1000. The benign refusal rates (2.8% to 8.5%) and MMLU (50.1 to 51.0) stay stable over the whole range. For K ≥ 5000, direct harmful ASR goes back up to between 6.0% and 6.7% and GSM8K drops to 28.0, so we use the middle value K = 1000.

(2) Standard deviations of the main results. We added the standard deviations to Table 2 of the revised manuscript (10 seeds, mean ± standard deviation). The gap between CAESAR and the strongest baseline (SFT) is much larger than the standard deviations: Overall is 64.65 ± 4.87 vs. 33.17 ± 3.45, the adversarial benign refusal rate is 4.59 ± 0.67 vs. 14.80 ± 3.23, the direct benign refusal rate is 7.19 ± 1.20 vs. 18.75 ± 1.76, and adversarial harmful ASR is 2.00 ± 0.30 vs. 7.35 ± 0.69.

(3) Comparison at matched levels. On WildJailbreak, the trade-off curves obtained by sweeping each of the four methods over a set of update budgets are shown in Figure 8 (Appendix 7.8) of the revised manuscript (see also point (3) of our response to W1). The CAESAR curve lies to the lower left of the baselines overall: at similar ASR, the benign refusal rate of CAESAR is clearly lower, and with adversarial harmful ASR on the x-axis, all 12 baseline configurations are dominated by some CAESAR configuration. So the improvement is not limited to the selected configuration. On MUSE-Book, SFT has lower Forget and ASR (0.3 / 1.5, compared with 2.3 / 12.7 for CAESAR), but its General drops from 84.3 to 49.6. After each metric is converted to an attainment ratio relative to the base model, the worst objective of SFT is 0.59, and that of CAESAR is 0.82.

W3 (knowledge removal or learned refusal)

Thank you for the valuable comments. We agree that low likelihood or ROUGE alone does not demonstrate complete removal of target knowledge from the model parameters. We would like to highlight that our goal herein is consistent with LLM approximate unlearning research (Eldan & Russinovich, 2023; Maini et al., 2024; Shi et al., 2025), which alters the model's output distribution and suppresses the likelihood of responses associated with the forget data. This operational notion of empirical forgetting is widely adopted in current approximate LLM-unlearning research. To reflect such empirical forgetting effects, we evaluate using multiple behavioral metrics, including ROUGE and prefilling ASR judged by an LLM judge. We deem provable and exact unlearning an important research direction, but it is out of the scope of this paper, which focuses on improving multi-objective alignment.

As a complement, we added an LLM-judge-based Forget evaluation: the judge decides whether a response gives the reference answer or an equivalent statement, without relying on lexical overlap (Table 15 in Appendix 7.14 of the revised manuscript). The Forget of CAESAR is 2.3 with ROUGE and 3.2 with the LLM judge, compared with 32.1 and 34.3 for the base model. The two measures lead to the same conclusion, and the ranking of the non-oracle methods is identical. We also adopted the reviewer's suggestion and rephrased the related conclusion in the revised manuscript as "reduced disclosure of the target knowledge under the evaluated conditions".

Thank you again for your valuable suggestions, which have helped strengthen our paper. We look forward to your further feedback.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer yfby,

Thank you again for your constructive review. As today is the last day of the author-reviewer discussion, we provide a short recap of your main concerns and our responses, including the new experiments added in the revision. Details are in our point-by-point replies.

1. Matched control with a shared teacher instruction (W1). Your concern: A control with the same reference answers, prefixes, loss, and budget is needed to isolate the contribution of objective-specific conditioning. Our response: We added this control experiment. A shared instruction carries both the "refuse" and the "answer" requirements, so its supervision conflicts across objectives. Overall drops from 64.65 to 30.64 and adversarial ASR rises from 2.00% to 25.34%, while utility is almost unchanged (Table 11).

2. Effective update strength as a confound (W1). Your concern: The smaller within-objective improvements of CAESAR in Figures 1-2 leave update strength as a possible explanation. Our response: In a new comparison at the same update magnitude, CAESAR has lower interference (+0.040 vs. +0.132 nats/token) and a larger average within-objective gain (-0.187 vs. -0.117). The reduction therefore does not come from smaller steps (Figures 1-2).

3. Per-objective K results, variability, and matched levels (W2). Your concern: The K sweep reported only Overall, the main results had no measure of variability, and the gains might hold only at the selected configuration. Our response: Figure 7 now shows every metric across K. Table 2 reports standard deviations over 10 seeds (Overall 64.65 ± 4.87 vs. 33.17 ± 3.45 for SFT). In our new trade-off sweep, all 12 baseline configurations are dominated by a CAESAR configuration (Figure 8).

4. Knowledge removal or learned refusal (W3). Your concern: The findings would be more appropriately framed as reduced disclosure under the evaluated conditions. Our response: We agree and adopted this framing, which is consistent with approximate unlearning research. We also added a new LLM-judge-based Forget evaluation, which gives the same method ranking (Table 15).

We would greatly appreciate your feedback on whether these points resolve your concerns, and we will respond promptly to any remaining questions while the discussion is open. If a reply today is not possible, we kindly ask that you consider these points and the revised manuscript in your final assessment.

Thank you for your time and for helping us improve the paper.

Reviewer_i4CgOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies multi-objective foundation model alignment from the perspective of supervision construction. The authors propose Caesar, a self-distillation framework that constructs objective-specific teacher distributions using task instructions and gold reference responses. The resulting supervision is distilled into the student through bidirectional Top-K logit matching.

Experiments on LLM unlearning and safety alignment report favorable trade-offs across competing objectives. Additional ablations investigate reference conditioning, distillation objectives, and limited-data settings.

Critical strengths

The paper highlights the importance of supervision construction in multi-objective alignment, complementing existing approaches that primarily focus on optimizing predefined objective functions. The reported results demonstrate encouraging empirical performance across unlearning and safety alignment.

Critical weaknesses
  1. Limited methodological novelty and scientific insight.

The proposed approach largely builds on established techniques in soft-label supervision, context distillation, and self-distillation. In particular, Learning by Distilling Context (Snell et al., 2022) already investigates transferring behaviors induced by privileged context into a model without that context. More closely related, DUET (Zhong et al., 2026) applies contextualized teacher distillation to LLM unlearning.

Although Caesar introduces objective-conditioned self-anchored supervision and bidirectional Top-K logit matching, the paper does not clearly establish a sufficiently distinct scientific insight beyond these existing approaches. Its potential contribution therefore depends heavily on demonstrating that the shared teacher construction provides a specific benefit for multi-objective compatibility.

  1. Insufficient justification for the claimed reduction in cross-objective interference.

The paper argues that objective-specific distributions derived from a shared base model naturally reduce cross-objective interference. However, sharing a teacher model does not inherently guarantee compatible optimization directions. As shown in Algorithm 1, different objectives are optimized through separate updates to shared parameters, so gradient conflicts may still arise.

Although Figures 2 and 5 show reduced interference and predictive divergence, these observations could also be explained by the regularization effect of soft supervision or more conservative parameter updates induced by self-distillation.

The paper includes comparisons with DUET and BiLD distillation losses in Appendix Table 4, which support the effectiveness of its loss design. However, these comparisons do not isolate whether the shared teacher construction itself improves cross-objective compatibility.

Consequently, the proposed mechanism remains insufficiently established.

  1. Insufficient evidence for genuinely improved multi-objective trade-offs.

The experimental comparisons do not adequately control for differences in optimization strength. For example, on MUSE-Book (Table 1), SFT achieves stronger forgetting (0.3 vs. 2.3) and lower ASR (1.5 vs. 12.7) than Caesar, while Caesar preserves substantially more general knowledge.

This raises the possibility that Caesar achieves a more conservative optimization solution rather than fundamentally reducing negative transfer between objectives.

Furthermore, the equally weighted Overall metric may obscure meaningful degradation in individual objectives, particularly those related to safety or forgetting. Comparisons at matched Forget or ASR levels, or Pareto trade-off curves across checkpoints, would provide more convincing evidence that Caesar improves the attainable multi-objective trade-off rather than merely selecting a different operating point.

Questions for authors

Q1. What is the specific scientific contribution beyond existing context-distillation and self-distillation methods?

In particular, what distinguishes the proposed shared self-anchored supervision from existing approaches such as DUET, beyond the choice of distillation loss and its application to multiple objectives?

Q2. Does Caesar genuinely improve cross-objective compatibility, or does it primarily produce more conservative parameter updates?

Could the authors compare knowledge retention at matched Forget or ASR levels, or provide trade-off curves across existing training checkpoints? Additionally, gradient norms, parameter-update magnitudes, or pairwise gradient cosine similarities across objectives would help distinguish reduced task conflicts from weaker optimization.

5 comments
AuthorsOpenReview ↗

Response to W1

W1 (novelty and scientific insight)

We thank the reviewer for raising this valuable point and would like to clarify our contributions below.

(1) One of our contributions is identifying an important yet overlooked problem: the quality of the supervision signal plays a crucial role in mitigating the interference among objectives in multi-objective alignment (MOA) (pages 1-2, lines 48-55). Existing multi-objective methods handle conflicts mostly after the supervision is fixed, by reweighting losses or reconciling gradients, presuming that each objective already has a suitable learning target.

(2) CAESAR constructs interference-tolerant supervision with minimal resources. Our method is simple yet effective. It needs no reward model, no preference pairs, no separately trained teacher, and no per-objective loss weights. Prior work shares common ground with us in using contextualized distillation, but it focuses on task-specific performance. This paper deeply investigates the role of self-generated supervision among competing objectives. One central finding is that building a separate self teacher for each objective from the same base model directly improves compatibility among objectives. We added a matched control experiment to isolate this point: the reference answers, prefixes, loss, and training budget are kept unchanged, and only the per-objective teachers are replaced by a single teacher that carries all objectives (one shared instruction).

TeacherDirect ASR ↓Direct Refusal ↓Adversarial ASR ↓Adversarial Refusal ↓MMLU ↑GSM8K ↑Overall ↑
Per-objective teachers (CAESAR)3.697.192.004.5950.0932.0364.65
One teacher for all objectives (one shared instruction)4.6913.3325.348.5050.032.5030.64

Note: the metrics are the same as in Table 2 of the paper (%).

Utility is almost unchanged, but adversarial harmful ASR rises from 2.00% to 25.34%, and the direct benign refusal rate rises from 7.19% to 13.33%. When one teacher carries both the "refuse" and the "answer" requirements, the supervision it gives conflicts across objectives. Building a separate teacher for each objective avoids this conflict. This benefit also shows up directly in the interference: at matched update size, the average interference of CAESAR on objectives with the opposite decision is +0.040 nats/token, compared with +0.132 for SFT (see our response to W2).

In addition, the experimental results echo our main finding that the self teacher provides less conflicting supervision: when the self teacher is replaced by an external model (same size or larger), Overall drops to between 4.92 and 30.83.

(3) CAESAR achieves strong performance gains over baselines, and our analysis shows that the gains mainly come from the construction of the supervision. On MUSE-Book, the Overall of DUET is 5.52. Keeping the DUET loss and only replacing the supervision with per-objective, reference-conditioned self-anchored teachers gives 74.33. Switching to our bidirectional loss then gives 87.32 (Table 1 and Table 4). The isolation experiments above have been added to the revised manuscript.

AuthorsOpenReview ↗

Response to W2 (Part 1)

W2 (mechanism behind the reduced interference)

We thank the reviewer for this insightful question. We agree that a shared teacher does not guarantee that the optimization directions of different objectives are fully compatible, and gradient conflicts may still occur. We would like to highlight that the claim of this paper is that self-anchored supervision introduces less interference than one-hot supervision, and that it is complementary to gradient-level methods (lines 295-297). For the two alternative explanations raised by the reviewer, we added controlled experiments.

(1) The lower interference is not due to more conservative updates. We compare SFT and CAESAR checkpoints at the same distance from the initialization (∥θ−θ0∥=1.35\lVert\theta-\theta_0\rVert=1.35, 3 seeds), using the interference metric Δ(i→j)\Delta(i\to j) from Figure 1 (nats/token; lower is better). The full results are given as heatmaps in the updated Figure 1 (SFT) and Figure 2 (CAESAR) of the revised manuscript.

Among the 8 objective pairs with opposite decisions (one requires refusal and the other requires answering), CAESAR has lower interference on 7, and the average drops from +0.132 to +0.040 nats/token. The improvement on the objective itself is not smaller (the diagonal average is -0.187, compared with -0.117 for SFT).

(2) The lower interference is not a uniform regularization effect. The soft teacher keeps probability support on alternative phrasings that are also plausible besides the gold demonstration, while hard labels suppress them (see the example in Figure 4). Furthermore, we ran three additional tests where candidates denote the non-gold tokens to which the teacher assigns a probability of at least 0.1:

Gold tokenCandidate tokenLow-probability control
Accuracy of the continuation after forcing this token (%)92.387.218.2
Change in log-prob after trainingValid candidatesInvalid candidatesLow-probability control
SFT-0.80-0.68-0.36
CAESAR+0.65+0.08+0.05

A candidate is valid if, after we force it and let the teacher continue, the whole response is automatically judged correct (for a harmful request, a refusal that the HarmBench classifier judges not harmful; for a benign request, an answer), and invalid otherwise. Low-probability controls are tokens ranked beyond 100 by the teacher at the same position. Whether a response is a refusal or an answer is decided by the same LLM judge as in our main evaluation, and all judgments are automatic, with no human annotation. The vast majority of the candidates are valid (87.2%). SFT suppresses valid candidates (0.80) more than the low-probability control tokens (0.36). CAESAR only raises the valid candidates and has almost no effect on invalid candidates and control tokens. If soft supervision were only uniform smoothing (e.g., label smoothing), the three types of tokens would be affected to a similar degree. In addition, at the same progress on the objective itself, reinforcing a candidate harms the opposite-decision objectives less than reinforcing gold, by 0.052 nats/token on average (21 of 26 pairs). When only the top-1 token of the teacher distribution is kept (K = 1), Overall is 31.19, comparable to 33.17 for SFT, and the advantage disappears.

AuthorsOpenReview ↗

Response to W2 (Part 2) and W3

W2 (mechanism behind the reduced interference), continued

(3) Theoretical grounding. Our analysis builds on the statistical view of distillation in Menon et al. (ICML 2021). Let h=(x,y<t)h=(x,y_{\lt t}) be the conditional context, q(⋅∣h)q(\cdot\mid h) the distribution of the self-anchored teacher, and let gv=∇θ[−log⁡πθ(v∣h)]g_v=\nabla_\theta[-\log\pi_\theta(v\mid h)] be the hard-label gradient for token vv. The gradient of the soft target is the expectation of the hard-label gradients under qq, i.e., gˉ=Ev∼q[gv]\bar g=\mathbb E_{v\sim q}[g_v]. When the demonstrated token is one of the acceptable phrasings described by qq (yt∼qy_t\sim q), the SFT update is gyt=gˉ+ξg_{y_t}=\bar g+\xi, where E[ξ]=0\mathbb{E}[\xi]=0 and Cov(ξ)=Σq\mathrm{Cov}(\xi)=\Sigma_q. A second-order expansion of the loss LjL_j of another objective jj gives:

Eyt∼q[Lj(θ−ηgyt)]−Lj(θ−ηgˉ)=η22tr(HjΣq)+o(η2)\mathbb E_{y_t\sim q}\big[L_j(\theta-\eta g_{y_t})\big]-L_j(\theta-\eta\bar g)=\frac{\eta^2}{2}\mathrm{tr}(H_j\Sigma_q)+o(\eta^2)

In expectation, the hard label moves the parameters in the same direction as the soft target, but incurs an additional increase in LjL_j (non-negative when Hj⪰0H_j\succeq 0). This term arises from which phrasing the demonstration happened to use, and soft supervision removes it. At the logit level, the one-hot gradient on a non-gold token vv equals πθ(v∣h)\pi_\theta(v\mid h), so tokens the model already considers likely, which are often legitimate alternatives, are pushed down hardest. This matches the empirical finding above that SFT suppresses valid candidates the most. The analysis is summarized in Section 5.4 (pages 9-10), and the full derivation is in Appendix 7.9 (pages 21-23).

(4) The effect of the shared-teacher construction itself has been isolated separately. Please kindly see point (2) of our response to W1.

W3 (whether the trade-off truly improves)

We thank the reviewer for this suggestion and have added the requested analyses.

(1) Following the suggestion, we added trade-off curves. See Figure 8 (Appendix 7.8) of the revised manuscript. On WildJailbreak, CAESAR, SFT, GKD, and Lisa are each swept over a set of update budgets. The x-axis is the ASR on harmful requests and the y-axis is the refusal rate on benign requests, so the lower left is better. The CAESAR curve lies to the lower left of the baselines overall.

With adversarial harmful ASR on the x-axis, all 12 baseline configurations are dominated by some CAESAR configuration (neither metric is better). With direct harmful ASR on the x-axis, the number is 9, and the remaining 3 obtain a lower ASR at the cost of a benign refusal rate of 18% to 46%. No CAESAR configuration is dominated by a baseline. For example, when direct harmful ASR is around 3% to 4%, the direct / adversarial benign refusal rates of CAESAR are 7.19% / 4.59%, compared with 18.74% / 14.87% for SFT and 14.26% / 41.68% for Lisa.

(2) On MUSE-Book, CAESAR is also not a more conservative solution. It removes 93% of the Forget score of the base model (from 32.1 to 2.3) and 82% of the prefilling ASR (from 70.5 to 12.7), while General stays at 84.3. SFT has lower Forget and ASR, but its General drops to 49.6 and its MMLU drops from 60.8 to 50.2.

(3) Normalized and worst-objective aggregates. As in prior work (Jang et al., 2023; Maini et al., 2024; Zhong et al., 2026), we use Overall to make the comparison intuitive, and the main evidence is still the per-objective results (Tables 1 to 3). Following the suggestion, we added aggregates that do not allow objectives to offset each other: each metric is converted to an attainment ratio relative to the base model (1 means fully attained), and we then take the mean (Mean) and the worst objective (Worst).

MethodMUSE-Book (Mean / Worst)WMDP-Cyber (Mean / Worst)WildJailbreak (Mean / Worst)
Strongest baselineSFT: 0.85 / 0.59DUET: 0.82 / 0.59SFT: 0.89 / 0.84
CAESAR0.95 / 0.820.97 / 0.900.95 / 0.83

CAESAR has the highest mean attainment on all three benchmarks. Its worst objective is the highest on MUSE-Book and WMDP-Cyber, and is on par with SFT on WildJailbreak. Both aggregates are reported together with Overall in the revised manuscript (Appendix 7.15 and Table 16, pages 26-27, with a pointer in Section 5.1, line 337).

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their constructive feedback and for recognizing the strengths of our work: the perspective of supervision construction as a useful and under-explored lens (Reviewers i4Cg, UQ3D); a simple and practical method (Reviewers yfby, 6So6, UQ3D); favorable trade-offs across unlearning and safety alignment (Reviewers i4Cg, 6So6, UQ3D); notable data efficiency (Reviewers yfby, UQ3D); and careful evaluation with informative ablations and interference diagnostics (Reviewers yfby, UQ3D). To address the concerns shared by several reviewers, we have added the following experiments and analyses. All of them have been added to the revised manuscript (changes are marked in blue):

  1. Interference comparison at matched update size (Reviewers i4Cg, yfby, 6So6, UQ3D): Experiments show that CAESAR still shows substantially lower cross-objective interference than SFT. This indicates that the reduction comes from how CAESAR resolves conflicts among objectives, not from taking smaller steps.

  2. Safety and compliance trade-off curves (Reviewers i4Cg, yfby, UQ3D): By sweeping each method’s trade-off parameter, we show that CAESAR improves the Pareto frontier compared with the baselines, and does not simply select a different operating point on the same frontier.

  3. Isolation experiments on supervision construction (Reviewers i4Cg, yfby, 6So6, UQ3D): Results conclude that building a separate self-teacher for each objective clearly provides better supervision than using one teacher for all objectives (matched distillation control; Reviewers i4Cg, yfby). We also added ablations with short generic instructions (Reviewer UQ3D) and with external teachers of the same size and larger (Reviewer 6So6).

  4. Mechanism analysis and theoretical grounding (Reviewer i4Cg): Token-level analysis shows that our soft teacher preserves valid alternative phrasings alongside the gold token: forcing a teacher-supported candidate still yields a correct continuation 87.2% of the time, vs. 92.3% for gold. Hard-label SFT suppresses these alternatives, while CAESAR selectively reinforces them and leaves invalid and low-probability tokens nearly unchanged, so the effect is not uniform regularization. Keeping only the teacher’s top-1 token (K = 1) removes CAESAR’s advantage entirely. Theoretically, following the statistical view of distillation (Menon et al., 2021), the hard-label update equals the soft-target update plus a zero-mean noise term determined by the particular phrasing the demonstration happened to use. This noise leaves the expected update direction unchanged but adds a second-order increase in the loss of the other objectives, which soft supervision removes.

  5. Additional evaluation: we have added the following experiments: (1) robustness evaluation under out-of-distribution attacks (Reviewer 6So6); (2) over-refusal evaluation on OR-Bench (Reviewer 6So6); (3) an LLM-judge-based forget rate as a complementary metric (Reviewers yfby, 6So6); (4) the base Tulu-2-7B model before alignment as a reference row in Table 2 (Reviewer UQ3D); (5) a conservative SFT baseline with a KL penalty to the base model (Reviewer 6So6); and (6) finer-grained multi-objective analysis, including per-objective Top-K results and standard deviations (Reviewer yfby), and normalized and worst-objective aggregates (Reviewers i4Cg, UQ3D). All new results are consistent with our original findings and further strengthen our main claims.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewers,

Thank you again for your constructive feedback. We have posted our point-by-point responses and added the supporting experiments and analyses to the revised manuscript, with changes marked in blue. We would appreciate your feedback on whether these additions address your concerns and would be happy to clarify any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.