- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how to improve language models' adherence to requirements in system prompts. It proposes Contrastive Self-Distillation for System-prompt adherence (CoSSy), which learns from the model's own prediction shift under system conditioning. CoSSy computes a contrastive teacher distribution at fixed, self-generated response prefixes and distills it with reverse KL into a student that still receives the system prompt. The trained student generates with ordinary inference, removing the extra contrastive model pass required by decoding-time guidance. Experiments show improved adherence across the tested models and transfer from single-turn training to multi-turn evaluation. My primary concern is the incremental significance of the proposed method relative to existing contrastive guidance and distillation ideas.
Critical strengths
- A practical supervision design. CoSSy converts the model's own response to system conditioning into training supervision, improving adherence and transferring to multi-turn evaluation (Tables 1–2). It needs no external target responses, preference labels or stronger teacher, and retains system conditioning with ordinary inference (§3.2).
Critical weaknesses
-
The contribution remains incremental. Previous literature has explored related ideas (CAD; SCOTT). CoSSy's combination of fixed self-generated prefixes, token-distribution supervision and a system-conditioned student is distinct, but I find its additional contribution limited. The hard-target contrastive ablation captures much of the gain in the reported Qwen3-4B IFBench setting (Table 6). This supports contrastive supervision while limiting the demonstrated added value of full-distribution matching. The gains and ordinary inference remain useful, but I find the training recipe insufficiently significant for acceptance.
-
The causal explanation remains unestablished. Table 4 compares movement toward different targets from different initial distances, so larger JSD reduction does not identify the cause of the advantage over external-teacher distillation. Direct departure-from-Base measurements on each model's own responses, alignment and fitted-strength analyses, and fit-quality/sensitivity checks provide useful descriptive evidence. They do not separate shift magnitude, direction and target differences as causes of adherence gains. This limits the explanation while leaving the measured gains intact.
Questions for authors
-
Added value over simpler training. Are matched ordinary self-SFT results on the same sampled responses available? Was the hard-target contrastive objective evaluated beyond the reported Qwen3-4B IFBench ablation? Contrastive argmax targets and Self-RefKD's teacher-only reference context do not answer the ordinary self-SFT comparison.
-
Shift magnitude and target differences. Beyond the departure, alignment and effective-strength analyses, what evidence separates requested shift magnitude from target direction and initial distance in the Teacher-KD comparison? If unavailable, does the proposed explanation remain a descriptive association?
-
Presentation suggestion. Figure 1's small labels and equations make the contrastive teacher construction, fixed prefixes and student training difficult to follow at normal page size, although the training/inference distinction itself is understandable.
10 comments
Response to Reviewer Z3ub [2/6]
[W1-1] The contribution remains incremental relative to existing contrastive guidance and distillation (continued)
Lastly, the proposed method not only improves efficiency but also improves the system-prompt adherence capability itself, overall outperforming even CAD (Tables 1 and 2), and it does so “without external supervision”. This is particularly important because supervision from a stronger model is often costly and even infeasible for frontier models. Moreover, our experiments (Tables 1 and 2) show that CoSSy can even outperform the baselines supervised by a much larger (80B) external teacher. While our original experiments already include a 14B-scale model, to further demonstrate this potential, we additionally conduct new experiments on Qwen2.5-72B and consistently observe that CoSSy remains effective even at this scale. Our work also provides extensive analyses of how this improvement occurs (Figure 3 and Table 5; for Table 4, please see our response to [W2]).
Qwen2.5-72B results (Appendix C.7, Table 26)
IFBench
| Method | Loose ↑ | Strict ↑ |
|---|---|---|
| Base | 34.00 | 30.67 |
| Self-Ref KD | 36.33 | 32.67 |
| Self-DPO | 36.00 | 33.00 |
| CAD | 42.67 | 40.67 |
| CoSSy (Ours) | 45.00 | 42.33 |
IHEval
| Method | Rule Ali. ↑ | Rule Con. ↑ | Task Ali. ↑ | Task Con. ↑ | Safety Ali. ↑ | Safety Con. ↑ |
|---|---|---|---|---|---|---|
| Base | 81.56 | 22.58 | 84.28 | 49.07 | 96.46 | 26.18 |
| Self-Ref KD | 82.86 | 22.32 | 84.27 | 47.78 | 95.25 | 25.57 |
| Self-DPO | 86.38 | 33.04 | 84.26 | 62.18 | 97.21 | 36.69 |
| CAD | 87.11 | 38.28 | 83.27 | 60.83 | 92.38 | 40.98 |
| CoSSy (Ours) | 87.86 | 44.66 | 84.40 | 68.19 | 99.41 | 52.56 |
SysBench, PL, and SR
| Method | SysBench CSR ↑ | SysBench ISR ↑ | SysBench SSR ↑ | PL ASR ↓ | SR Harm. ↓ |
|---|---|---|---|---|---|
| Base | 77.02 | 58.16 | 34.80 | 25.00 | 2.84 |
| Self-Ref KD | 76.41 | 56.92 | 35.60 | 25.51 | 2.88 |
| Self-DPO | 78.68 | 60.92 | 38.12 | 18.88 | 3.15 |
| CAD | 78.98 | 61.92 | 36.68 | 30.10 | 1.56 |
| CoSSy (Ours) | 80.46 | 64.76 | 40.08 | 20.92 | 1.20 |
Overall, improving system-prompt adherence is becoming an increasingly important problem for mitigating the risk of misalignment in LLMs, and we believe that our work fills a real gap that has not been explored before.
Response to Reviewer Z3ub [3/6]
[W1-2] Broader evaluation with full-distribution matching and hard-target contrastive supervision
We appreciate the reviewer raising this point. First, we would like to clarify that the hard target is itself derived from our contrastive target (i.e., the argmax of Eq. 3), so its result supports that the contrast is the main source of the gain. More importantly, reverse KL achieves consistently higher adherence than the hard target in multi-turn interactions. In addition, matching the full distribution with reverse KL additionally transfers how the contrast reshapes the rest of the distribution, as we show below.
#1. Extension Table 6 to include three models and SysBench
To directly address the reviewer’s concern, we repeated the objective comparison in Table 6 with three training runs and extended it to the three models and to SysBench. The updated Table 6 below reports the mean and standard deviation over the three runs on Qwen3-4B.
| Qwen3-4B (updated Table 6) | IFBench Strict ↑ | SysBench CSR ↑ |
|---|---|---|
| Base | 22.89 ± 0.51 | 73.98 ± 0.93 |
| CoSSy (reverse KL) | 36.00 ± 0.67 | 77.57 ± 1.17 |
| with forward KL | 33.67 ± 1.45 | 75.94 ± 0.79 |
| with cross-entropy (hard target) | 31.78 ± 2.99 | 75.73 ± 0.87 |
| student w/o system | 25.33 ± 7.05 | 67.49 ± 8.31 |
As shown in the updated Table 6, reverse KL clearly outperforms the hard target on both IFBench Strict (36.00 ± 0.67 vs. 31.78 ± 2.99) and SysBench CSR (77.57 ± 1.17 vs. 75.73 ± 0.87), in each of the three runs. We further report the three-model average below, where we average the three models within each run and then compute the mean and sample standard deviation across the three runs.
| Model Avg. | IFBench Loose | IFBench Strict | SysBench CSR | SysBench ISR | SysBench SSR |
|---|---|---|---|---|---|
| Base | 26.04 ± 0.30 | 22.52 ± 0.33 | 64.21 ± 0.34 | 43.46 ± 0.81 | 24.06 ± 0.92 |
| CoSSy (reverse KL) | 37.89 ± 0.57 | 33.70 ± 0.55 | 69.42 ± 0.63 | 51.06 ± 1.10 | 27.92 ± 1.25 |
| with forward KL | 38.11 ± 0.84 | 33.19 ± 0.97 | 65.54 ± 0.56 | 47.34 ± 0.72 | 25.41 ± 0.95 |
| with cross-entropy | 37.96 ± 1.10 | 32.89 ± 1.11 | 65.57 ± 0.47 | 47.14 ± 1.11 | 25.48 ± 0.88 |
| student w/o system | 28.44 ± 3.73 | 24.44 ± 3.77 | 51.74 ± 5.10 | 34.82 ± 5.60 | 19.08 ± 4.16 |
The advantage of reverse KL appears clearly in multi-turn adherence. While reverse KL and the hard target perform similarly on IFBench Strict (33.70 vs. 32.89), reverse KL is higher on SysBench on all three models (by 3.9 points in CSR, 3.9 in ISR, and 2.4 in SSR on average). Thus the gain of the hard target on single-turn instructions does not fully carry over to multi-turn interactions. This advantage also extends beyond Table 6: in our additional comparison in response to [Q1], CoSSy outperforms the hard target on most metrics.
#2. What matching the full distribution adds
To further examine whether each objective transfers the contrastive target as intended, we applied the measures of Table 5 to the students trained with each objective on the IFBench and SysBench Base responses of all three models. The table below shows Qwen3-4B (for full results, see Table 24).
| Objective (Qwen3-4B) | IFBench: System JSD ↑ | IFBench: Align. ↑ | SysBench: System JSD ↑ | SysBench: Align. ↑ |
|---|---|---|---|---|
| CoSSy (reverse KL) | .217 | +.69 | .261 | +.69 |
| with forward KL | .174 | −.23 | .227 | −.37 |
| with cross-entropy (hard target) | .180 | −.20 | .231 | −.32 |
Although the three objectives are trained toward the same contrastive target, only reverse KL changes the student in the way the system prompt does. With forward KL and the hard target, the sensitivity to the system prompt increases less (System JSD of .174–.180 vs. .217 on IFBench and .227–.231 vs. .261 on SysBench), and their change does not follow the shift induced by the system prompt (alignment of −.20 to −.37 vs. +.69). The same pattern holds on Llama-3.2-3B and Qwen2.5-14B, where reverse KL also gives the highest System JSD and by far the highest alignment (+.57 to +.76 vs. −.17 to +.18). Thus, full-distribution matching with reverse KL adds a consistent benefit on top of the contrast itself.
In the revision, we attribute the gain on IFBench mainly to the contrast and justify reverse KL by the SysBench results together with this analysis (Appendix C.5).
Response to Reviewer Z3ub [1/6]
Dear Reviewer Z3ub,
We appreciate your review with thoughtful comments. We have carefully considered each of your questions and provide detailed responses below.
[W1-1] The contribution remains incremental relative to existing contrastive guidance and distillation
We first thank the reviewer for recognizing our empirical contribution, including both the performance gain and the inference efficiency. We believe our core contribution lies exactly here: the new idea of “internalization to follow”, which completely removes the decoding-time overhead of contrastive decoding while preserving or improving its effectiveness, and we consider this a significant contribution to this field. Regarding the reviewer’s remaining concern about the significance of the training recipe itself, we would like to highlight that, among the components the reviewer lists, the “system-conditioned student” is not a minor design choice: it is essential for empirical success because it changes what is internalized. To the best of our knowledge, this idea has not been explored before, as also noted by Reviewer QiBb.
As the reviewer pointed out and as we also noted in the paper, CAD is already well established and has recently started to be applied to improve system-prompt adherence in LLMs. However, its inherent computational overhead, i.e., two forward passes and two KV caches for every generated token, often becomes a significant bottleneck for deploying it in practice; hence, some heuristics have been explored to reduce this cost. For example, SafeDecoding [1] applies contrastive decoding only to the first few tokens and then switches to conventional decoding, considering the importance of the starting tokens in the jailbreak setup. Instead of such a task-specific heuristic, we propose a simpler and more principled solution that removes this limitation by internalizing the contrastive distribution via self-distillation.
More importantly, such internalization does not simply stem from the distillation itself. In fact, how to distill matters significantly, and the key to the success is conditioning the student on the system prompt. While this modification looks quite simple, it completely changes what is internalized: from mimicking the behavior under specific system prompts (context distillation [2], which distills without conditioning the student) to general system-prompt adherence. The ablation in updated Table 6 empirically supports this idea: without the system prompt, the student does not reliably improve over Base (IFBench strict: 25.33 vs. 22.89 for Base and 36.00 for CoSSy). In other words, simply applying the standard context-distillation recipe to the contrastive target does not work, and identifying the formulation that does is a core part of our contribution. In this context, our work is also distinguished with the pointed SCOTT; SCOTT also combines contrastive decoding with distillation, but the contrast is used to obtain better rationales from a larger teacher, so what the student internalizes is the teacher's rationales rather than how to use a given context.
(Continued in Part 2/6.)