acceptodds
Under review as a conference paper at ICLR 2027

One Teacher, Many Tongues: Deficit-Aware Safety Distillation for Multilingual Alignment

15 views

Abstract

Multilingual safety alignment must reduce harmful responses across languages while preserving reasoning ability. Achieving this goal requires allocating on-policy safety-distillation supervision across both response positions and semantically equivalent language instances. We introduce DASD (Deficit-Aware Safety Distillation), a normalized token-level KL objective that allocates safety supervision along both axes. Adaptive prefix-tail weighting selects a response-specific prefix from cumulative divergence mass and gradually decays, rather than discards, tail supervision. English-anchored deficit calibration upweights target-language instances whose prefix-weighted divergence exceeds that of their paired English anchor. Both weights use predictions from a frozen, safety-conditioned copy of the student model, requiring neither an additional model nor a separate training stage. Across Qwen3 scales on MultiJail and PKU-SafeRLHF, DASD achieves the lowest average ASR among baselines for scale-benchmark pairs, while maintaining general capability on MMMLU and MGSM. Our code is available at https://anonymous.4open.science/r/12170F476.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multilingual Safety, On-Policy Self-Distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YauBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Deficit-Aware Safety Distillation (DASD) for multilingual safety alignment. DASD combines adaptive prefix-tail weighting, which emphasizes response prefixes with concentrated divergence while retaining decayed supervision on later tokens, with English-anchored calibration, which upweights target-language examples showing greater divergence than their English counterparts. Experiments on Qwen3 models of different scales show reduced attack success rates on MultiJail and PKU-SafeRLHF, improved worst-language safety, and better general-capability retention than the MSD baseline. Component ablations examine the contributions of both weighting mechanisms, and additional analyses investigate supervision allocation over response positions and generalization across different training-language sets.

Critical strengths

The proposed modifications is intuitive and clearly described. Its two components directly address supervision imbalance across response positions and languages, and the resulting objective is simple to implement. Their effectiveness is supported by reasonably controlled component ablations and evaluations across multiple experiment settings.

Critical weaknesses
  1. Several core design choices are heuristic and insufficiently justified. The overall comparison does not cleanly isolate the source of the large gains. For example, the mixed-KL formulation is not adequately motivated or ablated.
  2. The evaluation does not establish whether the safety gains come at the cost of excessive refusal. DASD is trained and evaluated for safety primarily on harmful queries, where refusing is sufficient to obtain a low ASR. MMMLU and MGSM measure general knowledge and reasoning, but do not test helpfulness on benign or safety-adjacent requests.
Questions for authors

See Critical Weaknesses.

2 comments

Thank you for the questions on design choices and benign-request behavior. Supplementary results use Qwen3-4B.

W1 / Q1: Design choices and sources of gains

DASD couples positional allocation with calibration of language-instance deficits. Section C1 of the General Response explains its differences from the underlying OPSD recipe and MSD's token-local weighting. Section C2 locates the original analyses, and Section C3 reports the new matched controls.

In brief, cumulative teacher–student divergence selects an adaptive prefix, while a decayed tail preserves later supervision. We then compare each position-weighted response with its semantically paired English response and increase the weight of instances with a positive relative deficit. The same positional weights define the deficit and the final normalized loss, connecting the two allocation stages without an additional trainable model.

We adds matched weighting and semantic-pairing controls in C3, plus the tail-decay control in the response to Reviewer cVtd, Q1/W1.

The mixed-KL choice now has a direct ablation. Its equal-weight mixture was defined in §3.2 (p. 3, lines 149–154); the new controls are in C3. Mean ASRs on MultiJail/PKU-SafeRLHF are 3.86%/4.53% for forward KL, 5.08%/6.51% for reverse KL, and 3.76%/4.13% for the mixture, which performs best among these choices on both benchmarks.

W2: Unclear trade-off between safety gains and over-refusal on benign requests

Our original evaluation followed the multilingual safety–capability design used by MPO and MSD: harmful-query safety evaluation together with multilingual knowledge and reasoning tests (MMMLU/MGSM). We agree that these capability tests do not directly establish helpfulness on benign sensitive requests.

Safety and over-refusal are a well-studied trade-off: stronger safeguards can also reject legitimate requests, as documented by OKTest and XSTest. Following your suggestion, we added over-refusal evaluation on OKTest and XSTest-safe in EN/ZH/AR/BN/KO/JA. We use GPT-4o to translate both datasets and to judge whether each model output refuses the benign request.

We conducted additional evaluations on benign samples. All three aligned methods below receive the same four-language benign augmentation. Rates are averaged over the six evaluation languages; lower is better.

ModelOKTest Refusal (%)XSTest-safe Refusal (%)
Raw Qwen3-4B5.2812.60
OPSA9.8316.47
MSD7.5618.13
DASD6.6116.67

DASD has less over-refusal on average across the two benchmarks. Giving the two benchmark means equal weight yields 11.64% for DASD, compared with 13.15% for OPSA and 12.85% for MSD.

Dear Reviewer YauB:

Thank you for your thoughtful review and for appreciating the method’s intuitive design, simple implementation, and supporting component ablations.

To address your questions about the sources of the gains, we have clarified the connection between the two weighting mechanisms and added controlled weighting and semantic-pairing comparisons, together with a direct comparison of forward, reverse, and mixed KL objectives. We have also provided six-language over-refusal evaluations on benign and safety-adjacent requests to examine the safety–helpfulness trade-off more directly.

We understand you may have other reviewing commitments. With fewer than 24 hours left, we would be grateful if you could revisit our rebuttal and share any remaining concerns or suggestions, which we would gladly address.

Best Regards,

The Authors

Reviewer_cVtdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Deficit-Aware Safety Distillation (DASD) to address inconsistent safety across languages. It proposes a new adaptive prefix–tail weighting scheme within an existing self-distillation framework. Across Qwen3-1.7B, 4B, and 8B, DASD achieves the lowest average attack success rates among the tested methods on MultiJail and PKU-SafeRLHF. It also preserves more knowledge and reasoning performance than MSD, although performance still drops relative to the original model.

Critical strengths
  • The paper compares with a good set of baselines and show empirical success against them.
Critical weaknesses
  • The motivation for the design of adaptive prefix-tail weighting is unclear and the design of unit weight then exponential tail weighting seems arbitrary, and introduces multiple hyperparameters.
  • As MSD already includes adaptive token weighting, and OPSA already supplies much of the underlying training recipe, the method’s overall novel contribution is limited, to the specific combination of adaptive prefix–tail weighting and English-relative instance weighting.
  • I find the paper is overall difficult to read.
Questions for authors
  • How much DASD's hyperparameter choices affect its performance, and how sensitive is the best performance depending on the hyperparameter?
2 comments

Thank you for the questions on design, hyperparameters, contribution, and presentation. Supplementary results use Qwen3-4B.

W1 / Q1: Unclear motivation for prefix–tail weighting and hyperparameter sensitivity

We compared adaptive and fixed prefix boundaries and analyzed allocation during training in the original submission. We now provide an upper-bound sensitivity study and a tail-decay control. Section 4.3 (p. 7, lines 354–356; Figure 2(b), p. 8) shows that the adaptive prefix reduces mean ASR from 8.21% to 6.83% and increases mean capability accuracy from 59.76% to 62.53% relative to a fixed 256-token prefix. Section 4.4 (p. 7, lines 361–377; Figure 4, p. 9) shows how supervision allocation evolves during training.

Design rationale. The mass target m = 0.7 selects the shortest prefix containing 70% of the response's divergence mass before clipping to 64–256 tokens. The 64-token minimum prevents the prioritized region from collapsing to a few tokens; the upper bound limits its extent. These settings are specified in §3.2 (p. 4, lines 197–208).

Raw-model prefix observations and the empirical target. We selected 50 responses from raw Qwen3-4B and measured divergence. We normalize divergence by each response's full-trajectory total, then average equally. The selected examples give the following cumulative mass:

Prefix tokens1282565121024
Mean cumulative mass (%)42.6977.0192.8198.15

We use m = 0.7 as the target to cover most of the divergence mass while retaining tail supervision; these examples illustrate the prefix divergence patterns targeted by this choice.

Training-time prefix observations. Figure 4 in the submission shows that, after an initial adjustment, the mean fraction of teacher–student divergence mass captured by the selected prefix for Qwen3-4B stabilizes at approximately 70%. Thus, the prioritized region continues to cover most of the divergence signal during actual training, while the tail retains decayed supervision.

Sensitivity to the prefix upper bound. With m = 0.7, λ = 1, we vary the prefix upper bound:

Prefix upper bound (tokens)MultiJail Mean ASR (%) ↓PKU-SafeRLHF Mean ASR (%) ↓
2563.764.13
5123.884.45
10242.954.76

Mean ASR varies by less than one percentage point on each benchmark across the tested upper bounds, indicating limited sensitivity in this range. Increasing the cap beyond 256 tokens provides no consistent benefit across both datasets, so we retain 256 as the default upper bound.

Additional tail-decay control. With m = 0.7 and the remaining settings fixed, disabling decay (λ = 0, tail weight 1) raises MultiJail mean ASR from 3.76% to 6.34%. Thai and Bengali ASRs rise from 4.44% and 9.84% to 13.34% and 16.80%, respectively. This supports reducing the tail's relative weight while retaining later supervision. This control tests sensitivity to tail decay.

W2: Novelty beyond existing self-distillation and token weighting

DASD couples response-level positional allocation with calibration of language-instance deficits against semantically paired English responses. This extends the underlying OPSD training procedure and differs from MSD's token-local DPSW weighting. We explain the inherited components and the new mechanism in Section C1 of the Shared Response, identify the original ablations in Section C2, and report the matched weighting controls in Section C3.

DASD achieves mean ASRs of 3.76%/4.13%, compared with 5.93%/4.76% for Prefix + DPSW on MultiJail/PKU-SafeRLHF. This directly supports the contribution of the proposed instance calibration within the shared distillation framework.

W3: The paper is difficult to follow

Thank you for the feedback on readability. As explained in Section C1 of the Shared Response, DASD first decides which response positions need more supervision, then which language instances need more supervision, and combines these weights in one loss. A deficit is the excess position-weighted divergence of a target-language response over its paired English response, after subtracting a margin.

The Method section follows this sequence:

  1. §3.1, Overview (p. 3, lines 125–134): introduces the two components and their connection.
  2. §3.2, Adaptive Prefix–Tail Weighting (pp. 3–4, lines 138–208): defines the shared student trajectory, teacher–student divergence, adaptive prefix, and decayed tail.
  3. §3.3, English-Anchored Positive-Deficit Calibration (pp. 4–5, lines 212–230): aggregates the weighted divergence and compares semantically paired language instances to obtain instance weights.
  4. §3.4, Unified Objective and Complexity (p. 5, lines 234–248): multiplies and normalizes the two weights, specifies the gradient flow, and explains the linear allocation cost.

We welcome further questions about any step that remains unclear.

Dear Reviewer cVtd:

Thank you for your time and constructive feedback, particularly your recognition of our broad baseline comparisons and empirical improvements.

In response to your concerns, we have clarified the motivation for adaptive prefix–tail weighting and how it works together with English-relative instance calibration. Our rebuttal adds prefix upper-bound sensitivity experiments, a tail-decay control, and raw-model divergence analysis. It also distinguishes the proposed components from the underlying self-distillation recipe and explains the method step by step to make the workflow easier to follow.

We appreciate your busy schedule. As the rebuttal closes in under 24 hours, could you kindly let us know whether any concerns remain? We would be happy to clarify them in the time available.

Best Regards,

The Authors

Reviewer_nNHoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies multilingual safety alignment through on-policy self-distillation. The authors propose DASD, which assigns greater weight to a response-dependent prefix and to language instances with larger divergence than their paired English queries. Both signals come from a frozen, safety-conditioned copy of the initial model. Experiments across Qwen3 sizes show improved safety on two multilingual benchmarks and better capability retention than MSD.

Critical strengths
  1. The method reuses a frozen copy of the initial model without requiring a stronger external teacher. Safety improvements appear across three model sizes and two benchmarks.

  2. The ablations show that the weighting rules improve over uniform distillation. Results for the weakest languages and general capability provide useful additional evidence.

Critical weaknesses
  1. Knowledge and mathematical accuracy do not show whether the model refuses legitimate sensitive requests. A benign request evaluation is needed to assess this part of the safety and helpfulness tradeoff.

  2. MSD already uses adaptive divergence weighting. The comparison also changes teacher context and KL settings, so the improvement over its weighting rule is not isolated.

  3. The ablation shows that instance weighting helps, but not why correct English pairing is needed. Shuffled anchors or absolute divergence weighting would help clarify this.

  4. The multilingual evaluations do not separately measure whether answers follow the requested language. This leaves one aspect of practical multilingual usefulness untested.

Questions for authors
  1. Does DASD retain useful answers on benign requests resembling harmful ones?

  2. Does DASD outperform DPSW when teacher context, KL support and training data are matched?

  3. Do correctly paired English anchors outperform shuffled anchors or absolute divergence weighting?

  4. How often do responses follow the requested language, particularly for Bengali and Arabic?

4 comments

Thank you for the questions on benign helpfulness, controlled comparisons, and the language used in model responses. Supplementary results use Qwen3-4B.

W1 / Q1: Missing evaluation of helpfulness on benign sensitive requests

Our original evaluation followed the multilingual safety–capability design used by MPO and MSD: harmful-query safety evaluation together with multilingual knowledge and reasoning tests (MMMLU/MGSM). We agree that these capability tests do not directly establish helpfulness on benign sensitive requests.

Safety and over-refusal are a well-studied trade-off: stronger safeguards can also reject legitimate requests, as documented by OKTest and XSTest. Following your suggestion, we added over-refusal evaluation on OKTest and XSTest-safe in EN/ZH/AR/BN/KO/JA. We use GPT-4o to translate both datasets and to judge whether each model output refuses the benign request.

We conducted additional evaluations on benign samples. All three aligned methods below receive the same four-language benign augmentation. Rates are averaged over the six evaluation languages; lower is better.

ModelOKTest Refusal (%)XSTest-safe Refusal (%)
Raw Qwen3-4B5.2812.60
OPSA9.8316.47
MSD7.5618.13
DASD6.6116.67

DASD has less over-refusal on average across the two benchmarks. Giving the two benchmark means equal weight yields 11.64% for DASD, compared with 13.15% for OPSA and 12.85% for MSD.

W2 / Q2: Gains over DPSW are not isolated from teacher and KL changes

With teacher context, KL support, training data, and the adaptive prefix fixed, DASD outperforms Prefix + DPSW on both benchmark means. Mean ASR decreases from 5.93% to 3.76% on MultiJail and from 4.76% to 4.13% on PKU-SafeRLHF.

Section C1 of the General Response explains the distinction between token-local DPSW and DASD's positional allocation with English-relative instance calibration. Section C2 identifies the original component ablations, and Section C3 gives the additional matched comparison requested here.

W3 / Q3: Unclear need for correctly paired English anchors

Thank you for suggesting these controls. Correct semantic pairing yields lower mean ASR than shuffled English anchors and absolute-divergence weighting on both benchmarks. Retaining the adaptive prefix gives:

Instance weightingMultiJail Mean ASR (%) ↓PKU-SafeRLHF Mean ASR (%) ↓
Shuffled English anchors7.488.58
Absolute divergence6.495.32
Correctly paired English anchors3.764.13

The improvement holds at the benchmark-average level and in the highlighted low-resource cases. Relative to shuffled anchors, correct pairing reduces Bengali ASR from 26.67% to 9.84% on MultiJail and from 23.34% to 8.33% on PKU-SafeRLHF. Relative to absolute weighting, PKU-SafeRLHF Bengali/Urdu ASRs decrease from 12.22%/17.78% to 8.33%/10.56%.

The paired reference compares the target language with English on the same semantic request, providing a relative language deficit beyond absolute divergence. Q4 further shows that Qwen3-4B's CoT is predominantly English across the evaluated input languages (100.00% on OKTest and 98.88% on XSTest-safe). This supports English as a reference for the reasoning process underlying multilingual responses. Our adaptive prefix emphasizes early response positions, and the same positional weights define the English-relative deficit. Together, the language observations and pairing controls support our design of coupling prefix-focused supervision with semantically paired English calibration.

W4 / Q4: Missing evaluation of adherence to the requested language

All Arabic and Bengali final answers from Qwen3-4B match the requested language in this evaluation, despite predominantly English reasoning. The table reports raw-model averages, separating the percentage of English chain-of-thought (CoT) traces from the percentage of final answers matching the input language.

Input LanguagesOKTest English CoT (%)OKTest Final Match (%)XSTest-safe English CoT (%)XSTest-safe Final Match (%)
Mean100.00100.0098.8899.92

Dear Reviewer nNHo:

Thank you for your careful review and for recognizing our safety gains without a stronger external teacher, along with the supporting ablations and results on weaker languages.

Following your suggestions, we have added six-language over-refusal evaluations, controlled weighting comparisons with teacher context, KL support, and training data held fixed, and tests of correctly paired versus shuffled English anchors and absolute-divergence weighting. We have also supplemented the rebuttal with an analysis separating reasoning language from final-answer language, including Arabic and Bengali.

We understand you may be busy. With less than 24 hours remaining before the rebuttal deadline, we would appreciate your feedback on any unresolved concerns and would be glad to address them.

Best Regards,

The Authors

Thank you for your rebuttal and additional experiments. The response addresses most of my concerns. However, for Q4, what I want to know is whether DASD training affects the language consistency rate.

Thank you for your follow-up and for acknowledging that our previous rebuttal clarified your concerns.

We are happy to address this question. We further analyzed DASD’s output trajectories. The results below show that DASD largely preserves Qwen3-4B’s language-use pattern alongside its safety improvements.

ModelOKTest English CoT (%)OKTest Final Match (%)XSTest-safe English CoT (%)XSTest-safe Final Match (%)
Qwen3-4B100.00100.0098.8899.92
DASD-4B99.9399.8799.9299.76

The same-model self-distillation leaves the languages used for reasoning and final answers largely unchanged, helping preserve the model’s original language habits.

Many thanks again for your efforts and time in reviewing our manuscript.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Shared concerns about method design and sources of gains

Thank you for the constructive feedback. This response addresses the shared novelty and design concerns in Reviewer nNHo's W2/Q2, Reviewer cVtd's W2, and Reviewer YauB's W1. Page and line references refer to the original submission. Results use Qwen3-4B.

C1. Design beyond existing self-distillation and token weighting

DASD couples response-level positional allocation with calibration of language-instance deficits. We build on the on-policy self-distillation (OPSD) recipe used by OPSA and compare with MSD. OPSA is the baseline name in our tables; OPSD denotes the underlying training procedure.

The OPSD recipe transfers safety-conditioned teacher behavior onto the student's own trajectories. MSD adds DPSW, which weights individual tokens using teacher confidence and the student's probability of the teacher-preferred token. MSD also supplies the English source query in the teacher context. DASD introduces two allocation mechanisms:

  1. Adaptive positional allocation. Cumulative teacher–student divergence selects a response-specific prefix. The prefix receives unit weight, and a smoothly decayed tail retains supervision at later positions. This uses the distribution of divergence across the response to determine where supervision should concentrate.
  2. English-relative instance calibration. For the same semantic request, we compare each target-language response's position-weighted divergence with its English counterpart. A positive deficit above a margin increases the target-language instance weight. English therefore provides a reference for allocating supervision across instances, rather than an additional query in the teacher prompt.

The same positional weights define both the instance deficit and the final normalized loss. This couples the two allocation decisions: language calibration measures the response regions that training emphasizes.

C2. Justification for allocation choices and hyperparameters

Component ablations. Section 4.3 (p. 7, lines 343–359) and Figure 2(b) (p. 8) compare five variants within the same DASD distillation setting. The reported means give equal weight to the two safety benchmarks and, separately, to the two capability benchmarks.

Original ablationMean ASR (%) ↓Mean Capability Accuracy (%) ↑
Uniform10.4362.72
Fixed-prefix (256)8.2159.76
Adaptive prefix–tail6.8362.53
Deficit-only4.5463.77
Full DASD3.9564.54

The adaptive-versus-fixed boundary comparison is explicitly discussed at p. 7, lines 354–356; the joint-component result appears at lines 357–359. These comparisons establish the contribution of the allocation components within the shared training framework.

Prefix allocation and parameter-related analysis. Section 4.4 (p. 7, lines 361–377) reports prefix lengths, selected response fractions, and captured divergence mass, with the trajectory audit in Figure 3 (p. 8) and training curves in Figure 4 (p. 9). For Qwen3-4B, the selected prefix averages 188.07 tokens; allocation stabilizes after the first 100 steps. The mass target, length bounds, decay rate, and tail floor are specified in §3.2 (p. 4, lines 197–208).

C3. Sources of gains and motivation for mixed KL

Matched weighting and semantic pairing. To isolate the weighting mechanism, we fix the teacher context, distillation KL support, and training data, and retain the same adaptive prefix. Prefix + DPSW replaces the English-relative instance calibration with MSD's token weighting. The other controls replace the correctly paired English reference with shuffled anchors or absolute-divergence weighting.

Weighting configurationMultiJail ASR (%) ↓PKU-SafeRLHF ASR (%) ↓
Prefix + DPSW5.934.76
Shuffled English anchors7.488.58
Absolute divergence6.495.32
Full DASD, correct English pairing3.764.13

DASD achieves lower mean ASR on both benchmarks under these controls. Relative to Prefix + DPSW, Bengali ASR also decreases from 17.78% to 9.84% on MultiJail and from 16.11% to 8.33% on PKU-SafeRLHF. The shuffled-anchor and absolute-divergence controls support the value of calibrating against the same semantic request in English. Thus, the allocation gains remain when the teacher context and KL settings are held fixed.

KL direction. We now add the following direction controls with teacher context, divergence support, training data, and allocation components fixed:

DivergenceMultiJail ASR (%) ↓PKU-SafeRLHF ASR (%) ↓
Forward KL3.864.53
Reverse KL5.086.51
Mixed KL3.764.13

The mixture has the lowest mean ASR among these choices on both benchmarks.

AuthorsOpenReview ↗

A Sincere Invitation to Review Our Rebuttal

Dear Reviewers,

Thank you for your time and constructive feedback. As the discussion period draws to a close, we sincerely invite you to review our rebuttal, supplementary experiments and analyses. We have also updated the manuscript PDF and anonymous code repository.

Our responses include:

  1. Over-refusal evaluations on OKTest and XSTest-safe across six languages. (Replies to nNHo, W1/Q1; YauB, W2.)
  2. Method clarifications covering distinctions from existing baselines, the coupled weighting mechanisms, deficit definition, and method workflow. (Shared response, C1; reply to cVtd, W2–W3.)
  3. Controlled comparisons of token weighting, correctly paired versus shuffled English anchors, absolute-divergence weighting, and forward/reverse/mixed KL. (Shared response, C3; replies to nNHo, W2/Q2–W3/Q3; YauB, W1/Q1.)
  4. Prefix-design analyses, including design rationale, a review of existing component ablations and training dynamics, plus new raw-model divergence analysis, prefix-cap sensitivity tests, and a tail-decay control. (Shared response, C2; reply to cVtd, W1/Q1.)
  5. Language-adherence analysis separating reasoning language from final-answer language, including Arabic and Bengali. (Reply to nNHo, W3/Q3–W4/Q4.)

We would greatly appreciate your feedback and welcome any further questions. Thank you again for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.