- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper frames foundation-model alignment as multi-objective optimization and argues that cross-objective interference stems partly from the supervision targets themselves, not only from how losses are aggregated. The proposed method, CAESAR, works as follows:
- For each objective , the frozen base model is conditioned on an objective-specific instruction and the gold response as privileged context.
- The resulting distribution serves as a soft teacher.
- This teacher is distilled into the student along gold-response prefixes using a bidirectional top- Smooth-L1 loss on raw logits, applied over both teacher-selected and student-selected supports (Eq. 5).
- Each minibatch is drawn from a single objective, so no loss weights are tuned.
A pilot study on WildJailbreak shows that one-batch SFT updates create cross-objective NLL interference (Fig. 1), which is reduced when the SFT target is replaced by the self-anchored teacher (Fig. 2, Fig. 5).
Experiments cover two unlearning benchmarks and one safety benchmark:
- MUSE-Book (Llama-3.2-3B) and WMDP-Cyber (Zephyr-7B), each with four objectives: forgetting, general retention, neighbor retention and prefilling robustness.
- WildJailbreak safety alignment (Tulu-2-7B), with direct/adversarial harmful and benign requests.
With 10 seeds and matched tuning budgets, CAESAR obtains the best aggregate "Overall" score in all three settings (Tables 1–3), with margins of +36.8, +44.9 and +31.5 points over the strongest baseline. Ablations cover reference conditioning, on-policy prefixes, alternative distillation losses, top-, and limited or imbalanced data (Tables 4–7).
Critical strengths
1. A simple, practical recipe with consistently strong results. CAESAR needs no reward model, no preference pairs, no separate teacher and no per-objective loss weighting. The method is easy to adopt, and the evaluation is reasonably careful: 10 seeds, a matched hyperparameter-search budget across methods, and validation-based checkpoint selection (Section 5.1). Gains appear across two unlearning benchmarks and one safety benchmark on three base models. The low-data and imbalanced-data results are especially notable. For example, under benign scarcity, adversarial benign refusal is 23.4% for CAESAR vs. 78.1% for SFT and 70.3% for Lisa, at equal adversarial ASR to Lisa (Table 7). These results suggest that soft self-generated targets are genuinely more data-efficient than one-hot demonstrations for refusal/compliance trade-offs.
2. Supervision construction is a useful and under-explored lens on interference. Most multi-objective alignment work (PCGrad, NGDiff, MOLLM) acts on gradients or weights after the targets are fixed. The paper instead asks how each objective's target distribution affects the others, and the per-cell interference matrix (Figs. 1–2) is an informative diagnostic. Fig. 4 is a concrete illustration of why soft targets help: "assist" and "help" both yield refusals, but SFT rewards only one of them.
Critical weaknesses
1. The interference analysis is confounded by update magnitude, and the method's novelty over context/self-distillation is limited.
- Smaller interference may just reflect smaller updates. The self-anchored teacher is the base model itself, so distilling toward it moves the student much less than an SFT step toward one-hot targets. The diagonal of the interference matrix confirms this. Within-objective NLL gains are also smaller under CAESAR for three of four objectives: −0.637 vs. −1.479, −1.425 vs. −2.076, and −0.878 vs. −1.289 (Fig. 2 vs. Fig. 1). The reduced off-diagonal interference may therefore reflect a smaller effective step rather than better cross-objective compatibility.
- The divergence measurement is circular. Fig. 5 measures KL to a fixed objective- teacher, which is the same base model under another prompt. A student distilled from that model family will trivially stay closer to it.
A fair diagnostic would compare students at matched within-objective gain or matched divergence from . Without it, the central claim that self-anchored supervision is "the dominant factor in reducing cross-objective interference" (Contribution 1) is not established.
On novelty, the method combines context distillation (Askell et al., 2021; Snell et al., 2022) with privileged-information self-distillation of the OPSD/SDFT type. It is closest to DUET (Zhong et al., 2026), which already uses prompt-steered self-supervision for unlearning. The incremental elements are per-objective prompts, reference conditioning and the Smooth-L1 logit loss. Related work in which models learn from spec-conditioned versions of themselves, such as deliberative alignment (Guan et al., 2024), should also be discussed.
2. The comparison likely favors CAESAR through prompt engineering and an aggregate metric that does not match the paper's own motivation.
- Teacher instructions encode benchmark-specific knowledge. The MUSE prompts (Appendix 8.1) are long, hand-tuned and benchmark-aware. For example, the prefilling prompt enumerates the exact neighbor works to keep answering (The Tales of Beedle the Bard, Fantastic Beasts, The Lord of the Rings). This injects knowledge of the evaluation's neighbor and forget boundaries into the supervision, which baselines trained on the same examples do not receive. The paper does not say how these instructions were developed or whether test data was consulted. Notably, "w/o ref. answer" already beats every baseline by a wide margin, which suggests the instructions carry much of the gain.
- Several baselines look degraded. GA increases Forget on MUSE-Book (36.9 vs. 32.1 for the base model; Table 1). SafeDecoding, SmoothLLM and Global Teacher drop MMLU to 28.0/27.5/34.4 and GSM8K to as low as 10.0 (Table 2), and Table 2 has no base Tulu-2-7B row against which to judge these drops.
- The aggregate rewards large swings, not balance. "Overall" is an unweighted sum of raw changes in heterogeneous metrics (ROUGE, accuracy, ASR, MMLU), so large swings on one axis dominate; on MUSE-Book, CAESAR's ASR drop alone contributes about 58 points. The introduction argues that reliable alignment "cannot tolerate failure on any single objective", which calls for a worst-objective or normalized metric rather than a sum.
The headline margins of 31–45 points therefore overstate the evidence, and it is unclear how much of the improvement survives with generic instructions and properly tuned baselines.
Questions for authors
- Can you report the Fig. 1/2 interference comparison at matched within-objective improvement, or at matched KL from ? For example, scale the SFT learning rate so that equals CAESAR's, and then compare the off-diagonal cells.
- How were the teacher instructions in Appendix 8 developed: on training data, validation data, or after inspecting test results? Do the works named in the MUSE prefilling prompt overlap with the neighbor or forget evaluation sets? How does CAESAR perform with a short, generic instruction per objective that names no benchmark entities?
- Please add the base Tulu-2-7B row to Table 2. Why do SafeDecoding, SmoothLLM and Global Teacher lose up to 30 MMLU points? Why does GA fail to reduce Forget on MUSE-Book? Were these baselines tuned with the same validation budget?
- Do the method rankings in Tables 1–3 hold under a normalized or worst-objective aggregate (e.g., the minimum per-objective improvement relative to the base model), which matches the stated motivation better than the unweighted sum?
- Beyond per-objective prompts and the Smooth-L1 loss, what distinguishes CAESAR from DUET and from standard context distillation? In particular, does DUET with your per-objective prompts close most of the gap?
6 comments
Response to W1 (Part 2)
W1 (novelty)
Thank you for pointing out these related works. We have added a discussion of Askell et al. (2021), SDFT, and deliberative alignment (Guan et al., 2024) to the Related Work of the revised manuscript. We would like to highlight the following two points.
(1) One of our contributions is identifying an important yet overlooked problem (pages 1-2, lines 48-55). Some of the prior work above shares common ground with us in using contextualized distillation or self-distillation, but it focuses on task-specific performance. The key difference lies in the problem we study: what causes cross-objective interference in multi-objective alignment, and how to construct supervision to mitigate it? This question leads to findings that are not covered by prior work on contextualized distillation:
- Interference originates partly from the supervision target itself. Under controlled diagnostics at matched update magnitude, one-hot supervision induces substantially more interference on opposite-decision objectives than self-anchored supervision (+0.132 vs. +0.040 nats/token; Section 3.2 and Figures 1-2, pages 3-4), and our analysis traces this to hard labels suppressing valid alternative phrasings (Section 5.4, pages 9-10, and Appendix 7.9).
- How the teacher is constructed determines the interference between conflicting objectives. A single teacher that encodes all requirements, as in DUET, gives conflicting supervision across objectives. In a matched control that changes only this choice, merging the per-objective teachers into one reduces Overall on WildJailbreak from 64.65 to 30.64 and raises adversarial ASR from 2.00% to 25.34% (Section 5.4, page 10, and Table 11 in Appendix 7.10).
- Anchoring the teacher to the student's own base model matters. Our additional experiments show that, replacing the self-anchored teacher with an external model of the same size or larger reduces Overall to between 4.92 and 30.83 (Section 5.4, page 10, and Table 12 in Appendix 7.11).
(2) Regarding Q5: giving DUET our per-objective teachers does close most of the gap, which is exactly bridged by this paper. On MUSE-Book, the Overall of DUET is 5.52. Keeping the DUET loss and only replacing the supervision with per-objective, reference-conditioned self-anchored teachers gives 74.33, and the full CAESAR gives 87.32 (Table 1 and Table 4). About 84% of the gap is closed by the construction of the supervision, and the rest comes from the bidirectional loss. Each part of the supervision construction is also isolated separately (WildJailbreak, where the full CAESAR is at 64.65). Replacing the per-objective instructions with one shared instruction gives an Overall of 30.64. Replacing the self teacher with an external teacher gives between 4.92 and 30.83. Removing the reference answer gives 46.76 (Table 11 in Appendix 7.10).
Response to W2 (Part 1)
W2 (teacher instructions)
Thank you for the valuable comments.
(1) How the instructions were written. All instructions were written only from the task definitions, the training set, and the validation set, without consulting test data or test results. The works named in the MUSE prefilling instruction are the works covered by the neighbor-retention training data. All baselines receive the same forget / retain boundary in the form of training examples, and the instruction only states this boundary in natural language. The neighbor evaluation set targets the same works, and its questions do not overlap with the training questions. This explanation has been added to Appendix 8 of the revised manuscript.
(2) Short generic instructions. Following this valuable suggestion, on WildJailbreak we replaced the instruction of each objective with a short generic instruction that names no benchmark entity. The full results and the instruction texts are in Table 11 and Appendix 7.10 of the revised manuscript.
With short generic instructions, Overall is 39.11, which is still higher than all baselines in Table 2 (the highest is 33.17 for SFT), and MMLU and GSM8K stay unchanged. So the advantage of CAESAR does not depend on instructions written for the benchmark. The more specific the instruction is, the more accurate the supervision given by the teacher is, which is consistent with the claim of this paper that the quality of the supervision signal is the key. In addition, the original instructions for WMDP-Cyber and WildJailbreak already name no benchmark entity (Appendix 8.2 and 8.3).
(3) On "w/o ref. answer already beats every baseline". This is consistent with the claim of this paper: supervision constructed per objective plays a crucial role. But the gain does not come from the information contained in the instructions themselves. When all requirements are written into one instruction and the model is prompted directly (Global Teacher), Overall is 28.52 on MUSE-Book and -13.11 on WildJailbreak. Distilling with such a shared instruction also gives only 30.64 (Table 11). The gain comes from constructing the supervision separately for each objective.
W2 (baselines)
Thank you for the suggestion. The base Tulu-2-7B row has been added to Table 2 of the revised manuscript. Relative to the base model, CAESAR lowers direct / adversarial harmful ASR from 21.88% / 60.57% to 3.69% / 2.00%, the benign refusal rates are almost unchanged, and MMLU drops by 2.3 points.
(1) The MMLU drop of SafeDecoding, SmoothLLM, and Global Teacher. This comes from the conflict between the safety objectives and general capability. The safety interventions of these three methods (the predictions of a safety expert, input perturbation, and a goal-prioritization prompt) do not distinguish which objective a request belongs to, so they harm the general capability of the model while lowering ASR. This conflict is exactly the problem this paper addresses: CAESAR constructs the supervision separately for each objective, and its MMLU is only 2.3 points lower than that of the base model.
(2) GA. All methods use the same criterion to select checkpoints: the highest Overall on the validation set (Section 5.1). GA only maximizes the loss on the forget data, its training is unstable, and the degradation of general capability appears before forgetting. At the selected checkpoint, MMLU has already dropped from 60.8 to 36.45, while Forget has not decreased yet (36.9, compared with 32.1 for the base model). Under the same criterion, a checkpoint with lower Forget would not have a higher Overall, so it was not selected. A similar trade-off can also be seen in Figure 8 (Appendix 7.8) of the revised manuscript (WildJailbreak): when a baseline pushes the ASR on harmful requests lower, its benign refusal rate goes up.
(3) Tuning budget. All baselines use the same hyperparameter search budget and checkpoint selection criterion, and the method-specific weights are also within the budget (Section 5.1, lines 365-367).
Response to W1 (Part 1)
W1 (confound of update magnitude)
Thank you for the thoughtful comments. In Figures 1 and 2, each method used its own training configuration, and the update size was indeed not controlled. Following the suggestion, we redid the comparison under matched conditions: both methods are read out at the same displacement from the initial parameters (∥θ−θ0∥=1.35, the minimum of the final displacements over all runs, chosen without looking at NLL; 3 seeds). The metric is the same as in Figure 1. It is the change in NLL on held-out reference answers of each objective, Δ(i→j) (nats/token, lower is better), and it does not depend on any teacher.
The full results are given as heatmaps in the updated Figure 1 (SFT) and Figure 2 (CAESAR) of the revised manuscript.
(1) At matched update size, the cross-objective interference of CAESAR is still lower. Among the 8 objective pairs with opposite decisions (one requires refusal and the other requires answering), CAESAR is lower on 7, and the average drops from +0.132 to +0.040 nats/token. The average over all 12 off-diagonal cells is +0.005, compared with +0.079 for SFT.
(2) The improvement on the objective itself is not smaller. At matched update size, the diagonal average is -0.187, compared with -0.117 for SFT. CAESAR improves more on Benign and Adv. benign (-0.181 vs. -0.008, and -0.248 vs. -0.054), and slightly less on Harmful and Adv. harmful (-0.063 vs. -0.083, and -0.256 vs. -0.321).
(3) At the behavior level, comparing at similar objective performance gives the same conclusion. When direct harmful ASR is around 3% to 4%, the direct / adversarial benign refusal rates of CAESAR are 7.19% / 4.59%, compared with 18.74% / 14.87% for SFT and 14.26% / 41.68% for Lisa.
We have also changed "the dominant factor" in Contribution 1 to "a crucial factor", so that the statement matches the evidence.
W1 (divergence measure in Figure 5)
Thank you for pointing this out. We would like to clarify that in Figure 5 both students start from the same θ0 and are compared with the same fixed reference under the same questions and prefixes, and the CAESAR student was not trained with the teacher of objective j. We agree that this measure uses the self teacher as the reference. The main evidence of this paper is the NLL interference matrix, which does not depend on any teacher (Figures 1 and 2), and the two lead to the same conclusion.