acceptodds
Under review as a conference paper at ICLR 2027

Following by Contrasting: Contrastive Self-Distillation for System-Prompt Adherence

Abstract

System prompts specify the roles, constraints, and preferences that large language models (LLMs) should follow. However, models often fail to reliably respect these requirements, particularly when they conflict with user requests or learned behavioral tendencies. Existing approaches to improving system-prompt adherence commonly rely on external behavioral supervision or additional computation during decoding. To address this, we propose Contrastive Self-Distillation for System-prompt adherence (CoSSy), a self-distillation framework that learns from the model's own prediction shift under system conditioning. Specifically, CoSSy contrasts predictions with and without the system prompt, amplifies the resulting shift, and distills it into a student conditioned on the system prompt, thereby learning to more effectively use the provided system prompt. This eliminates the need for external responses or preference labels while retaining standard autoregressive decoding at inference time. Across three LLMs on general system-message following and challenging system-prompt adherence benchmarks, CoSSy consistently outperforms competitive baselines. For example, CoSSy improves over the base models by 8.6% relative on average on Multifaceted-Bench and by 49.7% on IFBench Strict. Our analyses further reveal that CoSSy internalizes the supervision as an adaptive, state-dependent contrast rather than a fixed decoding-time intervention.

Primary area:
foundation or frontier models, including LLMs
Keywords:
system-prompt following

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Z3ubOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to improve language models' adherence to requirements in system prompts. It proposes Contrastive Self-Distillation for System-prompt adherence (CoSSy), which learns from the model's own prediction shift under system conditioning. CoSSy computes a contrastive teacher distribution at fixed, self-generated response prefixes and distills it with reverse KL into a student that still receives the system prompt. The trained student generates with ordinary inference, removing the extra contrastive model pass required by decoding-time guidance. Experiments show improved adherence across the tested models and transfer from single-turn training to multi-turn evaluation. My primary concern is the incremental significance of the proposed method relative to existing contrastive guidance and distillation ideas.

Critical strengths
  1. A practical supervision design. CoSSy converts the model's own response to system conditioning into training supervision, improving adherence and transferring to multi-turn evaluation (Tables 1–2). It needs no external target responses, preference labels or stronger teacher, and retains system conditioning with ordinary inference (§3.2).
Critical weaknesses
  1. The contribution remains incremental. Previous literature has explored related ideas (CAD; SCOTT). CoSSy's combination of fixed self-generated prefixes, token-distribution supervision and a system-conditioned student is distinct, but I find its additional contribution limited. The hard-target contrastive ablation captures much of the gain in the reported Qwen3-4B IFBench setting (Table 6). This supports contrastive supervision while limiting the demonstrated added value of full-distribution matching. The gains and ordinary inference remain useful, but I find the training recipe insufficiently significant for acceptance.

  2. The causal explanation remains unestablished. Table 4 compares movement toward different targets from different initial distances, so larger JSD reduction does not identify the cause of the advantage over external-teacher distillation. Direct departure-from-Base measurements on each model's own responses, alignment and fitted-strength analyses, and fit-quality/sensitivity checks provide useful descriptive evidence. They do not separate shift magnitude, direction and target differences as causes of adherence gains. This limits the explanation while leaving the measured gains intact.

Questions for authors
  1. Added value over simpler training. Are matched ordinary self-SFT results on the same sampled responses available? Was the hard-target contrastive objective evaluated beyond the reported Qwen3-4B IFBench ablation? Contrastive argmax targets and Self-RefKD's teacher-only reference context do not answer the ordinary self-SFT comparison.

  2. Shift magnitude and target differences. Beyond the departure, alignment and effective-strength analyses, what evidence separates requested shift magnitude from target direction and initial distance in the Teacher-KD comparison? If unavailable, does the proposed explanation remain a descriptive association?

  3. Presentation suggestion. Figure 1's small labels and equations make the contrastive teacher construction, fixed prefixes and student training difficult to follow at normal page size, although the training/inference distinction itself is understandable.

10 comments
AuthorsOpenReview ↗

Response to Reviewer Z3ub [1/6]

Dear Reviewer Z3ub,

We appreciate your review with thoughtful comments. We have carefully considered each of your questions and provide detailed responses below.


[W1-1] The contribution remains incremental relative to existing contrastive guidance and distillation

We first thank the reviewer for recognizing our empirical contribution, including both the performance gain and the inference efficiency. We believe our core contribution lies exactly here: the new idea of “internalization to follow”, which completely removes the decoding-time overhead of contrastive decoding while preserving or improving its effectiveness, and we consider this a significant contribution to this field. Regarding the reviewer’s remaining concern about the significance of the training recipe itself, we would like to highlight that, among the components the reviewer lists, the “system-conditioned student” is not a minor design choice: it is essential for empirical success because it changes what is internalized. To the best of our knowledge, this idea has not been explored before, as also noted by Reviewer QiBb.

As the reviewer pointed out and as we also noted in the paper, CAD is already well established and has recently started to be applied to improve system-prompt adherence in LLMs. However, its inherent computational overhead, i.e., two forward passes and two KV caches for every generated token, often becomes a significant bottleneck for deploying it in practice; hence, some heuristics have been explored to reduce this cost. For example, SafeDecoding [1] applies contrastive decoding only to the first few tokens and then switches to conventional decoding, considering the importance of the starting tokens in the jailbreak setup. Instead of such a task-specific heuristic, we propose a simpler and more principled solution that removes this limitation by internalizing the contrastive distribution via self-distillation.

More importantly, such internalization does not simply stem from the distillation itself. In fact, how to distill matters significantly, and the key to the success is conditioning the student on the system prompt. While this modification looks quite simple, it completely changes what is internalized: from mimicking the behavior under specific system prompts (context distillation [2], which distills without conditioning the student) to general system-prompt adherence. The ablation in updated Table 6 empirically supports this idea: without the system prompt, the student does not reliably improve over Base (IFBench strict: 25.33 vs. 22.89 for Base and 36.00 for CoSSy). In other words, simply applying the standard context-distillation recipe to the contrastive target does not work, and identifying the formulation that does is a core part of our contribution. In this context, our work is also distinguished with the pointed SCOTT; SCOTT also combines contrastive decoding with distillation, but the contrast is used to obtain better rationales from a larger teacher, so what the student internalizes is the teacher's rationales rather than how to use a given context.

(Continued in Part 2/6.)

AuthorsOpenReview ↗

Response to Reviewer Z3ub [2/6]

[W1-1] The contribution remains incremental relative to existing contrastive guidance and distillation (continued)

Lastly, the proposed method not only improves efficiency but also improves the system-prompt adherence capability itself, overall outperforming even CAD (Tables 1 and 2), and it does so “without external supervision”. This is particularly important because supervision from a stronger model is often costly and even infeasible for frontier models. Moreover, our experiments (Tables 1 and 2) show that CoSSy can even outperform the baselines supervised by a much larger (80B) external teacher. While our original experiments already include a 14B-scale model, to further demonstrate this potential, we additionally conduct new experiments on Qwen2.5-72B and consistently observe that CoSSy remains effective even at this scale. Our work also provides extensive analyses of how this improvement occurs (Figure 3 and Table 5; for Table 4, please see our response to [W2]).

Qwen2.5-72B results (Appendix C.7, Table 26)

IFBench

MethodLoose ↑Strict ↑
Base34.0030.67
Self-Ref KD36.3332.67
Self-DPO36.0033.00
CAD42.6740.67
CoSSy (Ours)45.0042.33

IHEval

MethodRule Ali. ↑Rule Con. ↑Task Ali. ↑Task Con. ↑Safety Ali. ↑Safety Con. ↑
Base81.5622.5884.2849.0796.4626.18
Self-Ref KD82.8622.3284.2747.7895.2525.57
Self-DPO86.3833.0484.2662.1897.2136.69
CAD87.1138.2883.2760.8392.3840.98
CoSSy (Ours)87.8644.6684.4068.1999.4152.56

SysBench, PL, and SR

MethodSysBench CSR ↑SysBench ISR ↑SysBench SSR ↑PL ASR ↓SR Harm. ↓
Base77.0258.1634.8025.002.84
Self-Ref KD76.4156.9235.6025.512.88
Self-DPO78.6860.9238.1218.883.15
CAD78.9861.9236.6830.101.56
CoSSy (Ours)80.4664.7640.0820.921.20

Overall, improving system-prompt adherence is becoming an increasingly important problem for mitigating the risk of misalignment in LLMs, and we believe that our work fills a real gap that has not been explored before.

AuthorsOpenReview ↗

Response to Reviewer Z3ub [3/6]

[W1-2] Broader evaluation with full-distribution matching and hard-target contrastive supervision

We appreciate the reviewer raising this point. First, we would like to clarify that the hard target is itself derived from our contrastive target (i.e., the argmax of Eq. 3), so its result supports that the contrast is the main source of the gain. More importantly, reverse KL achieves consistently higher adherence than the hard target in multi-turn interactions. In addition, matching the full distribution with reverse KL additionally transfers how the contrast reshapes the rest of the distribution, as we show below.

#1. Extension Table 6 to include three models and SysBench

To directly address the reviewer’s concern, we repeated the objective comparison in Table 6 with three training runs and extended it to the three models and to SysBench. The updated Table 6 below reports the mean and standard deviation over the three runs on Qwen3-4B.

Qwen3-4B (updated Table 6)IFBench Strict ↑SysBench CSR ↑
Base22.89 ± 0.5173.98 ± 0.93
CoSSy (reverse KL)36.00 ± 0.6777.57 ± 1.17
with forward KL33.67 ± 1.4575.94 ± 0.79
with cross-entropy (hard target)31.78 ± 2.9975.73 ± 0.87
student w/o system25.33 ± 7.0567.49 ± 8.31

As shown in the updated Table 6, reverse KL clearly outperforms the hard target on both IFBench Strict (36.00 ± 0.67 vs. 31.78 ± 2.99) and SysBench CSR (77.57 ± 1.17 vs. 75.73 ± 0.87), in each of the three runs. We further report the three-model average below, where we average the three models within each run and then compute the mean and sample standard deviation across the three runs.

Model Avg.IFBench LooseIFBench StrictSysBench CSRSysBench ISRSysBench SSR
Base26.04 ± 0.3022.52 ± 0.3364.21 ± 0.3443.46 ± 0.8124.06 ± 0.92
CoSSy (reverse KL)37.89 ± 0.5733.70 ± 0.5569.42 ± 0.6351.06 ± 1.1027.92 ± 1.25
with forward KL38.11 ± 0.8433.19 ± 0.9765.54 ± 0.5647.34 ± 0.7225.41 ± 0.95
with cross-entropy37.96 ± 1.1032.89 ± 1.1165.57 ± 0.4747.14 ± 1.1125.48 ± 0.88
student w/o system28.44 ± 3.7324.44 ± 3.7751.74 ± 5.1034.82 ± 5.6019.08 ± 4.16

The advantage of reverse KL appears clearly in multi-turn adherence. While reverse KL and the hard target perform similarly on IFBench Strict (33.70 vs. 32.89), reverse KL is higher on SysBench on all three models (by 3.9 points in CSR, 3.9 in ISR, and 2.4 in SSR on average). Thus the gain of the hard target on single-turn instructions does not fully carry over to multi-turn interactions. This advantage also extends beyond Table 6: in our additional comparison in response to [Q1], CoSSy outperforms the hard target on most metrics.

#2. What matching the full distribution adds

To further examine whether each objective transfers the contrastive target as intended, we applied the measures of Table 5 to the students trained with each objective on the IFBench and SysBench Base responses of all three models. The table below shows Qwen3-4B (for full results, see Table 24).

Objective (Qwen3-4B)IFBench: System JSD ↑IFBench: Align. ↑SysBench: System JSD ↑SysBench: Align. ↑
CoSSy (reverse KL).217+.69.261+.69
with forward KL.174−.23.227−.37
with cross-entropy (hard target).180−.20.231−.32

Although the three objectives are trained toward the same contrastive target, only reverse KL changes the student in the way the system prompt does. With forward KL and the hard target, the sensitivity to the system prompt increases less (System JSD of .174–.180 vs. .217 on IFBench and .227–.231 vs. .261 on SysBench), and their change does not follow the shift induced by the system prompt (alignment of −.20 to −.37 vs. +.69). The same pattern holds on Llama-3.2-3B and Qwen2.5-14B, where reverse KL also gives the highest System JSD and by far the highest alignment (+.57 to +.76 vs. −.17 to +.18). Thus, full-distribution matching with reverse KL adds a consistent benefit on top of the contrast itself.

In the revision, we attribute the gain on IFBench mainly to the contrast and justify reverse KL by the SysBench results together with this analysis (Appendix C.5).

Reviewer_QiBbOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes CoSSy, a self-distillation method for system-prompt adherence. The observation is that contrastive decoding (e.g., CAD, which samples from softmax(z+ + β(z+ − z−)) at each step) improves adherence but doubles inference cost. CoSSy keeps the same contrast but uses it only as a training target: a frozen base model is teacher-forced on its own generated prefixes with and without the system prompt, the amplified logit shift defines a token-level teacher distribution, and a system-conditioned student matches it via reverse KL. No external responses or preference labels are needed, and inference is standard autoregressive decoding. Across three models and nine benchmarks, CoSSy gives the best average on Multifaceted-Bench, large gains on IFBench-strict and IHEval conflict settings, better multi-turn adherence than CAD, and transfers to context faithfulness on ConFiQA. The most interesting part is the analysis: TeacherKD fits its external teacher better than CoSSy fits its target, yet CoSSy induces a ~2.4x larger policy shift in the contrastive direction, and the internalized contrast is state-dependent (effective β varies across decoding positions) rather than a fixed-strength intervention baked into the weights.

Critical strengths
  • The central idea is simple, well-motivated, and fills a real gap: the contrast that makes CAD work is turned into dense token-level supervision from the model itself, so adherence improves with zero inference overhead and no external teacher. The framing against context distillation (why the student must keep the system prompt at training time, Table 6's w/o-system collapse to 27.33/18.33) is a genuinely clarifying point that I have not seen stated this way before.

  • The mechanism analysis is what elevates the paper above a standard "apply KD to X" contribution. The target-fit-versus-policy-shift decomposition (Table 4) gives a satisfying answer to why distilling a stronger teacher fails while distilling the model's own amplified shift succeeds; the sensitivity/direction decomposition (Table 5) and the β_eff analysis (Figure 3) support the claim that what is learned is an adaptive, state-dependent contrast rather than a frozen decoding rule. These analyses are also the most reusable part for other work on self-distillation.

Critical weaknesses
  • No variance estimates anywhere in the main results. All tables appear to be single runs. Several load-bearing comparisons sit at the size where this matters: reverse KL vs forward KL differs by 1.3 points on IFBench-strict (36.0 vs 34.67), CoSSy and CAD tie on Llama IFBench-strict (29.0), and the paper's claim of beating Self-DPO rests on relative margins of 4.5% averaged over subsets with unknown run-to-run noise. For a post-training paper with LoRA at 1600 steps, at least 3 seeds on the headline tables should be feasible, and the design ablation (Table 6) in particular should not be used to justify reverse KL without repeated runs.

  • A few evaluation and reporting choices weaken confidence in the numbers as presented. The contrast strength β is selected on a 20% dev split of Multifaceted-Bench, which is one of the reported benchmarks; the remaining 80% is a fair test, but the same β is then carried to all other benchmarks, and the paper should say whether per-model β transfers or was re-tuned per benchmark. Some summary statements also overreach the tables: "consistently improves safety" does not hold for IHEval safety-aligned on Qwen3-4B (64.72 to 57.35, a drop shared with most baselines but not discussed), and "maintains general capabilities" glosses over MMLU-Redux dropping 1.7 points. These are fixable with honest scoping, but as written they read as selective reporting.

Questions for authors
  • Can you report mean and standard deviation over at least 3 training seeds for Table 1/2 and the Table 6 objective comparison? Specifically, is reverse KL's advantage over forward KL stable across seeds? (Could change my assessment; not a new method.)

  • Was β=6/6/2 selected only on Multifaceted-Bench dev, and how sensitive are IFBench/IHEval/SysBench results to β chosen on a different validation source? A small transfer check would address the selection concern.

  • IHEval safety-aligned drops for CoSSy on Qwen3-4B (64.7 to 57.4). Do you see this as amplification of the base model's safety behavior under conflict, or a training artifact? The paper currently claims consistent safety improvement.

  • Self-DPO uses five candidate responses per prompt while CoSSy uses one; what is the comparison at matched rollout budget, and what would CoSSy's cost be with on-policy prefixes (your stated future work) at equal wall-clock?

5 comments
AuthorsOpenReview ↗

Response to Reviewer QiBb [1/3]

Dear Reviewer QiBb,

We appreciate your review with thoughtful comments. We have carefully considered each of your questions and provide detailed responses below.


[W1, Q1] Lack of variance estimates and robustness across training seeds

To directly address the reviewer's concern, we conducted two additional independent training runs for every training-based method. For no training methods (Base and CAD), we instead repeated the evaluation with three inference seeds. We also extended the objective comparison to SysBench and all three models. The updated Table 6 reports the Qwen3-4B results, with the full three-model comparison provided in Table 23. The revised Tables 1 and 2 include the Self-SFT baseline and report mean scores, with standard deviations provided in Appendix Tables 31 and 32. Table 6 reports both means and standard deviations.

#1. Main results (Tables 1 and 2)

As shown in the revised Tables 1 and 2, the results are consistent. CoSSy achieves a 4.3% higher relative average score than Self-DPO on Multifaceted-Bench across the three models (4.029 vs. 3.863). Compared with CAD, CoSSy achieves higher average adherence on both IFBench Strict (33.70 vs. 32.89) and SysBench (CSR of 69.42 vs. 65.10), without the additional forward pass at inference. This indicates that the improvements of CoSSy hold across training seeds.

#2. Reverse KL vs. forward KL (Table 6)

The advantage of reverse KL over forward KL remains consistent across the three training runs. The updated Table 6 below reports the mean and standard deviation for Qwen3-4B.

MethodIFBench Strict ↑SysBench CSR ↑
Base22.89 ± 0.5173.98 ± 0.93
CoSSy (reverse KL)36.00 ± 0.6777.57 ± 1.17
with forward KL33.67 ± 1.4575.94 ± 0.79
with cross-entropy31.78 ± 2.9975.73 ± 0.87
student w/o system25.33 ± 7.0567.49 ± 8.31

On Qwen3-4B, reverse KL achieves higher IFBench Strict accuracy (36.00 ± 0.67 vs. 33.67 ± 1.45) and SysBench CSR (77.57 ± 1.17 vs. 75.94 ± 0.79) averaged over the three runs.

These results support the stability of the advantage across training runs and our choice of reverse KL as the default objective.

We additionally report the three-model averages below, computed by averaging the three models within each run and then calculating the mean and sample standard deviation across the three runs. Table 23 in the revised manuscript provides the per-model results for Base and the three supervision objectives.

IFBench — Model Avg.

Model Avg.IFBench LooseIFBench Strict
Base26.04 ± 0.3022.52 ± 0.33
CoSSy (reverse KL)37.89 ± 0.5733.70 ± 0.55
with forward KL38.11 ± 0.8433.19 ± 0.97
with cross-entropy37.96 ± 1.1032.89 ± 1.11
student w/o system28.44 ± 3.7324.44 ± 3.77

SysBench — Model Avg.

Model Avg.SysBench CSRSysBench ISRSysBench SSR
Base64.21 ± 0.3443.46 ± 0.8124.06 ± 0.92
CoSSy (reverse KL)69.42 ± 0.6351.06 ± 1.1027.92 ± 1.25
with forward KL65.54 ± 0.5647.34 ± 0.7225.41 ± 0.95
with cross-entropy65.57 ± 0.4747.14 ± 1.1125.48 ± 0.88
student w/o system51.74 ± 5.1034.82 ± 5.6019.08 ± 4.16

Averaged over the three models, reverse KL is comparable to forward KL on IFBench and higher on all three SysBench metrics in each of the three runs (on average by 3.9 points in CSR, 3.7 in ISR, and 2.5 in SSR). These results support the stability of the advantage across training runs and our choice of reverse KL as the default objective. We provide further analysis of reverse KL in our response to Reviewer Z3ub [W1-2], showing that only reverse KL effectively changes the student in the direction induced by the system prompt.

AuthorsOpenReview ↗

Response to Reviewer QiBb [2/3]

[W2-1, Q2] Selection of contrast strength β and its transferability across benchmarks

We originally selected β using the Multifaceted-Bench validation set, choosing 6, 6, and 2 for Qwen3-4B, Llama-3.2-3B, and Qwen2.5-14B, respectively. As described in Appendix B.4 and Table 11, these values were then kept fixed across all other benchmarks, without benchmark-specific retuning. To check whether the validation source affects this choice, we re-analyzed the existing β-sweep outputs, selecting β on a fixed 20% split of each benchmark and evaluating each selection on the remaining 80%.

Selected β by validation source

Validation sourceQwen3-4BLlama-3.2-3BQwen2.5-14B
Multifaceted-Bench662
IFBench662
IHEval664
SysBench622

Held-out benchmark scores

Modelβ (selected by)IFBench ↑IHEval ↑SysBench ↑
Qwen3-4B6 (all four sources)35.8356.8061.66
Llama-3.2-3B6 (MF, IFBench, IHEval)32.5053.7935.00
2 (SysBench)31.8847.9336.98
Qwen2.5-14B2 (MF, IFBench, SysBench)38.3366.8056.48
4 (IHEval)38.1368.0654.24

Alternative validation sources select the same β in seven of nine model–source combinations. Changing β improves performance on the benchmark used for selection but lowers scores on the other benchmarks in the two cases where β is reselected. Therefore, the original β selection is supported by the majority of validation sources for all models. We have added this validation-source comparison to Appendix B.4, together with the experimental procedure.


[W2-2, Q3] Overstated claims regarding safety improvement and general capability preservation

First of all, we acknowledge that some summary statements in the original draft overreach the tables, including the two pointed statements about IHEval safety on Qwen3-4B and the general capability. To resolve this, we modify such statements in the revision as below:

  • Lines 319 - 323: we replace "consistently improves safety" with a scoped statement that reports the decrease on Qwen3-4B and add this explanation with the per-model safety results.
  • Lines 510 - 513: Considering the additional experiments requested by Reviewer UGHz, we replace "CoSSy maintains performance comparable to the Base model" with "CoSSy exhibits average performance drops of less than one point across the three models on MATH500 (74.80 to 73.87) and MMLU-Redux (78.20 to 77.27) while slightly improving on LiveCodeBench (37.64 to 37.85)."

Next, regarding the reviewer's question about potential sources of IHEval safety-aligned drop on Qwen3-4B, we conjecture that it is an amplification of the base model's own behavior rather than a training artifact. This is because (1) the training target of CoSSy is made by the base model itself on the given task, and (2) Cossy successfully improves IHEval safety-aligned accuracy for other models, which have better initial performance.

Specifically, in terms of IHEval safety-aligned accuracy, Qwen3-4B has the lowest Base score (65.07 vs. 74.32 for Llama-3.2-3B and 94.61 for Qwen2.5-14B), and CoSSy improves (74.32 → 81.29) or maintains (94.61 → 94.46) the score on the other two models. On Qwen3-4B, training on the same self-generated responses without the contrast (Self-SFT) does not decrease the score (65.55), while most other trained methods in Table 2 also fall below Base. The decrease thus reflects the limited capability of the base model on this split rather than training on these responses.

In addition, we observe that a more capable model benefits from the same amplification. We newly applied CoSSy to a much larger backbone, Qwen2.5-72B. CoSSy improves all evaluated safety measures over Base and achieves the best results among the compared methods on IHEval safety (99.41 and 52.56) and StrongREJECT (1.20).

Qwen2.5-72B

MethodIHEval Safety Ali. ↑IHEval Safety Con. ↑PL ASR ↓SR Harm. ↓
Base96.4626.1825.002.84
CAD92.3840.9830.101.56
Self-RefKD95.2525.5725.512.88
Self-DPO97.2136.6918.883.15
CoSSy99.4152.5620.921.20
AuthorsOpenReview ↗

Response to Reviewer QiBb [3/3]

[Q4] The comparison with Self-DPO at matched rollout budget, and CoSSy's cost with on-policy prefixes at equal wall-clock

Following the reviewer’s suggestion, we conducted additional experiments that matched Self-DPO’s generation budget at six responses per prompt (one chosen response and five rejected response candidates) and kept CoSSy training fixed at 1,600 updates. For CoSSy, we generated five additional responses and selected the one with the largest initial reverse KL between the base model and the contrastive target among those six (1 existing rollout with 5 additional rollouts) as the training prefix.

As shown in Table 28 in Appendix C.9 of the revised manuscript, at the same rollout budget, CoSSy scored higher than Self-DPO on MF, both IFBench metrics, all six IHEval metrics, and SysBench CSR and ISR, but performed worse on SysBench SSR and PurpleLlama. However, increasing CoSSy’s rollout budget from one to six responses per prompt did not consistently improve its performance, while it is effective for some benchmarks like IHEval-safety.

For Self-DPO and one-rollout CoSSy, values are mean ± sample standard deviation over three runs. Six-rollout CoSSy reports a single run. Bold indicates the best score for each metric.

MF and IFBench

MethodMF Avg. ↑IFBench Loose ↑IFBench Strict ↑
Self-DPO (6 rollouts)4.118 ± 0.00431.11 ± 1.5826.33 ± 0.67
CoSSy (1 rollout)4.341 ± 0.00438.44 ± 0.2036.00 ± 0.67
CoSSy (6 rollouts)4.36439.0034.33

IHEval

MethodRule Ali. ↑Rule Con. ↑Task Ali. ↑Task Con. ↑Safety Ali. ↑Safety Con. ↑
Self-DPO (6 rollouts)86.28 ± 0.6339.88 ± 0.3966.45 ± 6.1330.70 ± 2.4455.23 ± 4.3011.95 ± 0.78
CoSSy (1 rollout)89.02 ± 0.1651.77 ± 1.0774.56 ± 0.4939.43 ± 3.2959.82 ± 4.4412.72 ± 0.53
CoSSy (6 rollouts)89.5949.9675.4941.9867.0615.99

SysBench, PL, and SR

MethodSysBench CSR ↑SysBench ISR ↑SysBench SSR ↑PL ASR ↓SR Harm. ↓
Self-DPO (6 rollouts)75.27 ± 0.8056.57 ± 1.9034.85 ± 3.0133.67 ± 1.020.11 ± 0.18
CoSSy (1 rollout)77.57 ± 1.1760.89 ± 2.6636.84 ± 3.4934.86 ± 3.240.43 ± 0.37
CoSSy (6 rollouts)76.4658.6834.2839.290.00

Next, we compared offline and on-policy CoSSy under the same compute allocation and end-to-end time budget (Appendix C.10, Table 29). Timing includes model loading, generation, training, and final checkpoint saving, but excludes benchmark evaluation. Offline CoSSy completed 1,600 updates in 8h 09m 34s; on-policy CoSSy completed 358 updates within this budget. Each setting was evaluated in a single run.

MethodTime budgetActual timeUpdatesTraining examples
Offline CoSSy8h 09m 34s8h 09m 34s1,60051,200
On-policy CoSSy8h 09m 34s8h 06m 49s35811,456

As shown in Table 30 in the same appendix, compared with the offline baseline, on-policy CoSSy achieved comparable MF performance (4.348 vs. 4.343) and higher IFBench scores (Strict 37.67 vs. 36.00), but performed worse on all six IHEval metrics, all three SysBench metrics, PurpleLlama (ASR 50.00 vs. 31.12; lower is better), and StrongREJECT. Thus, on-policy CoSSy completed fewer updates and scored lower on most metrics within the same time budget, supporting our choice of offline training.

Offline scores come from a separate standard CoSSy training run. The timed offline run was used only to measure wall-clock cost. On-policy scores are from the time-budgeted training run. Each performance row reports a single run. Bold indicates the best score for each metric.

MF and IFBench

MethodMF Avg. ↑IFBench Loose ↑IFBench Strict ↑
Offline CoSSy4.34338.3336.00
On-policy CoSSy4.34839.6737.67

IHEval

MethodRule Ali. ↑Rule Con. ↑Task Ali. ↑Task Con. ↑Safety Ali. ↑Safety Con. ↑
Offline CoSSy89.2152.3774.8542.9557.3513.31
On-policy CoSSy87.9548.6274.4541.8656.8412.20

SysBench, PL, and SR

MethodSysBench CSR ↑SysBench ISR ↑SysBench SSR ↑PL ASR ↓SR Harm. ↓
Offline CoSSy78.8563.9240.4831.120.00
On-policy CoSSy76.3158.8435.0450.000.32

We hope that our response and the additional experiments have fully addressed your concerns.

If you have any further questions/concerns, please do not hesitate to let us know.

Thank you very much,

Authors

Reviewer_UGHzOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CoSSY to improve LLMs’ compliance with system prompts. CoSSy computes the logits of a frozen model both with and without the system prompt, amplifies their difference following the principle of context-aware decoding, and distills the resulting distribution into a student model using reverse KL divergence. Across Qwen3-4B, Llama-3.2-3B, and Qwen2.5-14B, it outperforms baselines like Self-DPO and teacher-based SFT/DPO on Multifaceted-Bench, IFBench, IHEval, and SysBench while preserving general capabilities.

Critical strengths
  • The method is simple and cheap. CoSSy needs only system/user pairs, one self-generated response per prompt, and two teacher-forced passes of the frozen model per example.
  • The experiment setup is comprehensive which include multiple datasets and model sizes.
  • The baselines are strong and matched well with the proposed method.
Critical weaknesses
  • Novelty is a bit incremental. The core idea is distilling a guidance-style contrastive distribution (CAD / classifier-free guidance) into model weights. CAD was already well established and this paper only adds additional distillation on top of CAD.
  • The claim that CoSSy works because it requests a larger policy shift (about 2.4× more JSD reduction) is confounded by the starting point. CoSSy’s target begins much farther from Base (JSD .215 vs. .126), so a larger absolute reduction is almost guaranteed.
  • The analyses after the experimental section is a bit narrow. All analyses only use Qwen3-4B on IFBench, and Teacher KD runs only on Qwen3-4B because of tokenizer matching. The ConFiQA and general-capability experiments also seem limited to a single model.
Questions for authors

Please refer to the weaknesses.

7 comments
AuthorsOpenReview ↗

Response to Reviewer UGHz [1/5]

Dear Reviewer UGHz,

We appreciate your review with thoughtful comments. We have carefully considered each of your questions and provide detailed responses below.


[W1] Novelty is a bit incremental

We first clarify that our core contribution is resolving the fundamental limitation of contrastive decoding and providing a new way to improve system-prompt adherence via the novel idea of “internalization to follow”, rather than simply combining a guidance-style contrastive distribution with distillation.

As the reviewer pointed out and as we also noted in the paper, CAD (like classifier-free guidance) is already well established and has recently started to be applied to improve system-prompt adherence in LLMs. However, its inherent computational overhead, i.e., two forward passes and two KV caches for every generated token, often becomes a significant bottleneck for deploying it in practice; hence, some heuristics have been explored to reduce this cost. For example, SafeDecoding [1] applies contrastive decoding only to the first few tokens and then switches to conventional decoding, considering the importance of the starting tokens in the jailbreak setup. Instead of such a task-specific heuristic, we propose a simpler and more principled solution that removes this limitation by internalizing the contrastive distribution via self-distillation.

More importantly, such internalization does not simply stem from the distillation itself. In fact, how to distill matters significantly, and the key to success is conditioning the student on the system prompt. While this modification looks quite simple, it completely changes what is internalized: from mimicking the behavior under specific system prompts (context distillation [2], which distills without conditioning the student) to general system-prompt adherence. The ablation in Table 6 empirically supports this idea: without the system prompt, the student does not reliably improve over Base (IFBench strict: 25.33 vs. 22.89 for Base and 36.00 for CoSSy). In other words, simply applying the standard context-distillation recipe to the contrastive target does not work, and identifying the formulation that does is a core part of our contribution.

Lastly, the proposed method not only improves efficiency but also improves the system-prompt adherence capability itself, overall outperforming even CAD (Tables 1 and 2), and it does so “without external supervision”. This is particularly important because supervision from a stronger model is often costly and even infeasible for frontier models. Moreover, our experiments (Tables 1 and 2) show that CoSSy can even outperform the baselines supervised by a much larger (80B) external teacher. While our original experiments already include a 14B-scale model, to further demonstrate this potential, we additionally conduct new experiments on Qwen2.5-72B and consistently observe that CoSSy remains effective even at this scale. Our work also provides extensive analyses of how this improvement occurs (Figure 3 and Table 5; for Table 4, please see our response to [W2]).

IFBench

MethodLoose ↑Strict ↑
Base34.0030.67
Self-Ref KD36.3332.67
Self-DPO36.0033.00
CAD42.6740.67
CoSSy (Ours)45.0042.33

IHEval

MethodRule Ali. ↑Rule Con. ↑Task Ali. ↑Task Con. ↑Safety Ali. ↑Safety Con. ↑
Base81.5622.5884.2849.0796.4626.18
Self-Ref KD82.8622.3284.2747.7895.2525.57
Self-DPO86.3833.0484.2662.1897.2136.69
CAD87.1138.2883.2760.8392.3840.98
CoSSy (Ours)87.8644.6684.4068.1999.4152.56

SysBench, PL, and SR

MethodSysBench CSR ↑SysBench ISR ↑SysBench SSR ↑PL ASR ↓SR Harm. ↓
Base77.0258.1634.8025.002.84
Self-Ref KD76.4156.9235.6025.512.88
Self-DPO78.6860.9238.1218.883.15
CAD78.9861.9236.6830.101.56
CoSSy (Ours)80.4664.7640.0820.921.20

Overall, improving system-prompt adherence is becoming an increasingly important problem for mitigating the risk of misalignment in LLMs, and we believe that our work fills a real gap that has not been explored before (as also highlighted by Reviewer QiBb).

AuthorsOpenReview ↗

Response to Reviewer UGHz [2/5]

[W2] The claim that CoSSy works because it requests a larger policy shift (about 2.4× more JSD reduction) is confounded by the starting point.

We thank the reviewer for this careful observation, and we acknowledge that the JSD reduction in Table 4 can be confounded by the difference in the starting point. We will remove this claim and the causal wording built on it, and revise Table 4 to report measures that are not confounded by the starting point. Nevertheless, we believe that the core message still holds, as shown in the following two parts.

  • The Top-20 overlap in the same table, which starts from an even lower value at Base but becomes higher at Trained with CoSSy compared to Teacher KD, and additionally measurement, Top-1 change, show that CoSSy moves the student considerably further toward its target than Teacher KD does. In addition, with Self-RefKD and Teacher CAD newly added to Tables 3,4 and 5, we also confirm that only the targets with the contrast shift the model toward the tokens favored by the system prompt.

  • A derivation shows that most of what the teacher target requests is unrelated to the system prompt.

#1. Revised Table 4

In the revision, Table 4 reports the existing Top-20 overlap with the target, which starts lower for CoSSy than for Teacher KD but is reversed after training. To examine how much each supervision changes the decisions of the student independently of its target, we newly measured Top-1 change, i.e., the fraction of positions where the top token of the trained student differs from that of Base before training [3]. For the extensive analysis, we also add the results of Self-RefKD, which takes its supervision from the model itself without the contrast, and Teacher CAD, which distills the same contrast computed from the logits of the external teacher, softmax(zT++β∗(zT+−zT−))\mathrm{softmax}(z_T^+ + \beta*(z_T^+ − z_T^-)), where zT+z_T^+ and zT−z_T^- are the logits of the teacher with and without the system prompt. The table below shows the updated Tables 3 and 4.

MethodIFBench Loose ↑Strict ↑SysBench CSR ↑ISR ↑SSR ↑
Base27.0023.0075.0557.8836.44
External Teacher38.6736.6782.7169.4846.28
CAD33.0031.0072.6356.4033.48
Teacher KD25.0022.3373.0255.3633.96
Teacher CAD36.3333.3377.9962.7239.64
Self-RefKD25.3322.6774.8757.6436.04
CoSSy (Ours)38.3336.0078.8563.9240.48
MethodBase ↑Trained ↑Δ ↑Top-1 Change (%)
Teacher KD.606.639+.03315.0
Teacher CAD.493.535+.04222.9
Self-RefKD.562.616+.0546.1
CoSSy (Ours).588.677+.08926.5

Despite starting from a lower overlap with its target than Teacher KD (.588 vs. .606), CoSSy ends with a higher one (.677 vs. .639) and changes the top token at more positions (26.5% vs. 15.0%). For the overlap, the gap does not depend on the choice of k: it holds for every k from 5 to 100, including k = 50, where the two targets start from the same overlap (.584) and the gains are +.089 and +.033 (Appendix C.2). Teacher CAD changes the top token at more positions than Teacher KD (22.9% vs. 15.0%), although it moves less toward its target than CoSSy (+.042). Self-RefKD also moves toward its target (+.054) but changes few decisions (6.1%), since its target already agrees with Base at about 85% of the positions.

Together, these results suggest that the larger movement of CoSSy is not simply explained by differences in the starting point, and that improvements in adherence may be more closely related to whether the target contains the contrast than to the source of supervision: only the two targets with the contrast, from the model itself (CoSSy) or from the teacher (Teacher CAD), improve adherence here (Table 3). Whether these changes follow the direction induced by the system prompt is examined in Table 5 (please see our response to [W3-1] for details).

AuthorsOpenReview ↗

Response to Reviewer UGHz [3/5]

[W2] The claim that CoSSy works because it requests a larger policy shift (about 2.4× more JSD reduction) is confounded by the starting point. (con’t)

#2. Derivation to understand why Teacher KD is insufficient

Let ss and uu denote the system prompt and the user message, and let pBp_B and pTp_T denote the next-token distributions of Base and the external teacher at the same response prefix (omitted for brevity). As in Eq. (3), let z+z^+ and z−z^- denote the logits of Base with and without the system prompt; the contrastive target of CoSSy is then q(β)=softmax(z++β(z+−z−))q(\beta) = \mathrm{softmax}(z^+ + \beta(z^+ - z^-)). For a model MM, PMIM(y;s)=log⁡pM(y∣s,u)−log⁡pM(y∣u)\mathrm{PMI}_M(y; s) = \log p_M(y \mid s, u) - \log p_M(y \mid u) measures how much the system prompt ss raises the probability of a token yy, and CAD amplifies this quantity [4]. The change that each target requests from Base can then be written as follows. For CoSSy, whose target q(β)q(\beta) amplifies the logit shift z+−z−z^+ - z^- induced by ss in Base,

log⁡q(β)(y)−log⁡pB(y∣s,u)=β⋅PMIB(y;s)+const,\log q(\beta)(y) - \log p_B(y \mid s, u) = \beta \cdot \mathrm{PMI}_B(y; s) + \mathrm{const},

i.e., the system effect of Base itself, amplified. For Teacher KD, whose target is pT(⋅∣s,u)p_T(\cdot \mid s, u),

log⁡pT(y∣s,u)−log⁡pB(y∣s,u)=[log⁡pT(y∣u)−log⁡pB(y∣u)]+[PMIT(y;s)−PMIB(y;s)],\log p_T(y \mid s, u) - \log p_B(y \mid s, u) = [\log p_T(y \mid u) - \log p_B(y \mid u)] + [\mathrm{PMI}_T(y; s) - \mathrm{PMI}_B(y; s)],

where the first term is the difference between the teacher and Base that exists without the system prompt, and the second term is the difference in how the two models use the system prompt (const does not depend on yy). Teacher CAD adds β∗⋅PMIT(y;s)\beta^* \cdot \mathrm{PMI}_T(y; s), the system effect of the teacher itself, to this request.

To see which term dominates on the training prefixes, we measure the JSD to the teacher distribution with and without the system prompt, where both the model and the teacher receive or do not receive ss.

JSD to external TeacherTraining +systemTraining −system
Base0.1260.125
Teacher KD0.0970.097
CoSSy0.1970.147

Base is almost as far from the teacher without the system prompt as with it, indicating that the first term dominates: most of what the teacher target requests is unrelated to the system prompt. Accordingly, Teacher KD becomes closer to the teacher by the same amount in both contexts, which is consistent with its unchanged System JSD in Table 5. CoSSy, in contrast, moves away from the teacher mainly when the system prompt is given (+.071 vs. +.022), i.e., its change is concentrated on how the system prompt is used.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers and AC

Dear reviewers and AC,

We appreciate you for taking the time and making the effort to examine our manuscript.

As the reviewers highlighted, we propose a simple and well-motivated method (UGHz, QiBb) that fills a real gap (QiBb) by turning the model's own response to the system prompt into dense token-level supervision (QiBb, Z3ub), while being cheap to train (UGHz), free of external target responses, preference labels, or a stronger teacher (QiBb, Z3ub), and free of additional inference overhead (QiBb, Z3ub). Our approach is supported by comprehensive experiments with strong and well-matched baselines (UGHz), gains that transfer from single-turn training to multi-turn (Z3ub), and mechanism analyses that are reusable (QiBb), with clear writing (QiBb).

We appreciate your constructive feedback on our manuscript. In response, we have meticulously updated and enhanced the manuscript, including the following additional discussions and experiments:

  • Mean and standard deviation over three independent training runs for all training-based methods (Section 4.1, Tables 1, 2, and 6, Appendix D, Tables 31 and 32)
  • Self-SFT baseline trained on the same self-generated responses (Section 4.1, Tables 1 and 2, Appendix B.3)
  • Clarified selection of the contrast strength and its robustness to the validation source (Section 4.1, Appendix B.4, Tables 11–13)
  • Scoped claims on CAD, safety, and general capability, with per-model safety analysis (Sections 4.2 and 4.3, Appendix C.8, Table 27)
  • Revised explanation of the advantage over Teacher KD without the starting-point confound, with Top-20 overlap, Top-1 change, Teacher CAD, and Self-RefKD (Sections 1, 4.3, and 5, Tables 3–5, Appendix C.2, Tables 14 and 15)
  • Extension of all analyses to three models and the multi-turn SysBench, including Teacher KD with MinED alignment (Section 4.3, Appendix C.2–C.4, Tables 16, 20, and 21, Figures 5 and 8)
  • Supervision objectives, including the hard target, on three models and SysBench, and what each objective transfers (Section 4.3, Appendix C.5, Tables 23 and 24)
  • Effect of the contrast strength averaged over three models, with per-model results (Section 4.3, Figure 4, Appendix C.5, Figure 9)
  • Context faithfulness and general capability on three models (Section 4.3, Tables 7 and 8, Appendix C.6, Table 25)
  • Results on a larger model, Qwen2.5-72B (Section 4.2, Appendix C.7, Table 26)
  • Comparison with Self-DPO at a matched rollout budget and the cost of on-policy prefixes (Section 4.2, Appendix C.9 and C.10, Tables 28–30)
  • Improved readability of the method overview (Figure 1)

For your convenience, revisions have been temporarily marked in green in the amended manuscript.

We sincerely believe that these updates may help us better deliver the benefits of the proposed CoSSy to the ICLR community.

Thank you very much,

Authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.