acceptodds
Under review as a conference paper at ICLR 2027

Dual-Vocabulary Language Model for Cross-tokenizer Distillation

Abstract

On-policy distillation (OPD) bridges teacher supervision and student behavior, but different teacher-student tokenizers introduce misalignment in both input tokenization (#1) and output logits (#2). Existing approaches address the former by matching same-text spans or converting tokens to bytes, often losing fine-grained token information or disrupting the native-token paradigm, while for the latter, strategies such as ranking, padding, or key-token selection retain only shared logit dimensions, resulting in much distribution loss. In this paper, we propose Dual-Vocabulary Language Model (DVLM), which replaces the teacher’s LM head with a new student-vocabulary projection head and obtains full-dimensional student logits (for #2). To support student tokens (for #1), it takes a Parallel-Tokenized Sequence (PTS) as input, which concatenates the original teacher-tokenized sequence and a re-tokenized sequence formed by independently converting each student token into a teacher-token group. To avoid inference inconsistency with the original teacher tokens, the Hybrid-Prefix Attention (HPA) further restricts re-tokenized groups to their corresponding teacher prefix and uses its last state as the aggregation of the original student-token representation for projection into the student vocabulary space. Similarly, via the combined use of PTS and HPA, the DVLM teacher can provide distribution-aligned supervision with the student’s input-tokenization and output-logit during OPD. Experimental results demonstrate that our DVLM teacher has a similar converged loss as the original teacher model and enables student models to improve performance across six reasoning tasks. Our code and data: bluehttps://anonymous.4open.science/r/DVLM-C0B7/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs, on-policy distillation, cross-tokenizer

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Xxp7OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the challenges of On-Policy Distillation (OPD) when the teacher and student models use different tokenizers, which causes misalignments in both input token sequences and output vocabulary spaces. To resolve this, the authors propose the Dual-Vocabulary Language Model (DVLM), a framework that adapts the teacher's architecture to directly process the student's tokenizations and output predictions within the student's vocabulary space.

Contributions To achieve this cross-vocabulary alignment, the authors introduce a Dual-Vocabulary Teacher that replaces the teacher's original LM head with a newly-trained student-specific projection layer. This structural adaptation is supported by a Parallel-Tokenized Sequence (PTS) input format and a Hybrid-Prefix Attention (HPA) masking mechanism to align input representation spaces during a two-stage training pipeline.

Validation The authors evaluate DVLM by distilling a Qwen-3-4B teacher into Llama-3.2-1B and OLMo-2-1B student models across standard downstream NLP benchmarks.

Critical strengths

The core concept of adapting the teacher model to directly output logits in the student's token space is an elegant and promising intuition that avoids heuristic vocabulary truncation. Additionally, the paper is well-written, with a clear presentation of the methodology and mathematical formulations.

Critical weaknesses
  1. High Methodological Overhead for Marginal Performance Gains and Missing Baselines While the goal of bridging the vocabulary gap is commendable, the empirical gains appear marginal compared to the substantial overhead incurred. Adapting the teacher requires modifying the LM head, altering the attention mechanism, and introducing an extra pre-training stage (Stage I). Despite these efforts, the student models in Table 1 only achieve a modest improvement of ~2.8 to 3.1 points, leaving an enormous ~35-point performance gap with the teacher virtually untouched. Furthermore, because the teacher model itself is substantially altered, the authors do not report a self-distillation baseline of the adapted teacher on the student's vocabulary, making it difficult to verify whether the teacher's original generative capabilities were degraded during adaptation.

  2. Extreme Sensitivity and Fragility Revealed by the Ablation Study The ablation results in Table 2 raise critical questions regarding the robustness and underlying mechanism of the proposed framework. Specifically, when re-tokenization is removed (w/o Re-token), the performance catastrophically drops to 14.45 and 13.36 on Llama and OLMo, respectively—falling substantially below even the naive base student models (20.43 and 21.85). This dramatic collapse indicates that the adapted teacher backbone cannot reliably project onto the student's vocabulary space on its own, suggesting that the Parallel-Tokenized Sequence acts more as a fragile data-engineering workaround to prevent model collapse rather than facilitating robust, generalizable representation alignment.

  3. Invasive Teacher Architecture Modifications and Heuristic Attention Constraints Unlike conventional non-intrusive alignment methods (e.g., span-level or token-free approaches) where the teacher remains fully frozen, DVLM heavily alters the teacher’s core architecture. The Hybrid-Prefix Attention (HPA) relies on a set of hand-crafted, heuristic attention-masking rules to coordinate token visibility between the parallel sequences. Such bespoke masking constraints not only introduce engineering friction when scaling or transferring to different model families and context lengths, but they also compromise the plug-and-play nature expected from modern knowledge distillation paradigms.

Questions for authors

To better evaluate the true effectiveness of the proposed Dual-Vocabulary Teacher, could you provide the evaluation results (e.g., perplexity or zero-shot performance on standard benchmarks) of the adapted teacher model itself after Stage I training?

Could you include stage I when analyzing complexity / runtime?

1 comment
AuthorsOpenReview ↗

Rebuttal to Reviewer Xxp7

We are grateful for your careful review, helpful comments and detailed questions that ensued.

For W1&&Q1 'high overhead' and 'the evaluation results of the adapted teacher':

  • We apologize for any confusion and would like to clarify that our method makes only limited modifications to the teacher model.
    • In Stage I, we train only the new LM head, which is partially initialized from the original teacher’s LM head. All other teacher parameters, including the input token embeddings and attention weights, remain frozen throughout training.
    • HPA simply masks selected tokens in the standard causal attention matrix. This design preserves a sufficiently long prefix in the teacher’s native tokenization to keep inference consistent with the original teacher’s training setup.
    • Table 7 reports the training times for both stages. With the GPU count held fixed, Stage I increases the training cost by only about 8% in GPU-hours for both teacher–student pairs.
  • Our new teacher encodes inputs using its original tokenizer and predicts tokens from the student’s vocabulary. The following experiments show that adaptation preserves the teacher’s original generative capabilities.
    • Already Done: The new-head teacher converges to a similar loss and achieves comparable BPB (Table 8), while matching the original teacher’s performance on four comprehension tasks (Figure 4(b)) and nearly matching it on two reasoning tasks (Table 9).
    • Extending the evaluation in Table 9, we find that the new teacher performs on par with the original teacher on two additional reasoning benchmarks.
Teacher headGSM8KMath-500Humaneval+CRUXEval
Original Qwen head87.7968.4082.9341.81
DVLM (Llama vocabulary)87.9266.8082.3241.73
DVLM (OLMo vocabulary)87.7768.6081.7142.00

For W2 'sensitivity and fragility':

  • PTS and HPA are tightly coupled. Changing either component in isolation prevents the new LM head from being adequately trained.
    • HPA enables PTS to preserve a sufficiently long prefix in the teacher’s native tokenization, keeping inference consistent with the original teacher’s training setup. This prefix yields causal representations of student tokens, which the new LM head learns to map to the student vocabulary.
    • ’w/o Re-token' ablation removes the retokenized student sequence from PTS and retains only the teacher prefix. Without the student-token representations, the new LM head cannot converge.
    • The resulting degradation therefore stems from removing a prerequisite for cross-tokenizer alignment.
  • To assess the role of PTS in the new LM head’s convergence, we evaluate 'w/o Re-token' teacher itself on loss, BPB, and performance across two comprehension and two generation tasks. The results below show that the PTS structure is crucial for learning to map representations from the frozen teacher backbone to the student vocabulary.
MethodlossBPBArc-CMMLU-algebraMath-500Humaneval+
DVLM(Llama)1.930.370.89420.5466.8082.32
w/o Re-token2.010.420.88700.5063.0776.02

For W3 'DVLM heavily alters‘ :

  • As we clarify before, our method makes only limited modifications to the teacher model.
    • In Stage I, we train only the new LM head, which is partially initialized from the original teacher’s LM head. All other teacher parameters, including the input token embeddings and attention weights, remain frozen throughout training.
    • Our method uses the existing Transformer blocks as implemented. HPA simply applies additional masking to the default causal attention matrix, requiring minimal implementation effort and preserving the underlying attention logic.
    • Because our method requires retraining only the LM head, it can be adapted to any teacher-token pairs.
  • Our design has a clear mechanistic rationale.
    • We design PTS to learn representations for tokens unique to the student vocabulary from inputs tokenized with the teacher’s tokenizer. HPA lets student tokens attend to the aligned teacher prefix while preventing future-token leakage, and preserves a sufficiently long prefix in the teacher’s native tokenization to keep inference consistent with the original teacher’s training setup.
    • The ablation in Table 2 and the additional experiments above show that each of these design choices is necessary for the new LM head to converge.

For Q2 'analyzing runtime':

    • Table 7 reports the training times for both stages. With the GPU count held fixed, Stage I increases the training cost by only about 8% in GPU-hours for both teacher–student pairs.
Reviewer_EjwyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DVLM for on-policy distillation between models with different tokenizers. It freezes the teacher backbone and trains a new output head over the student's vocabulary. A parallel-tokenized sequence combines the teacher's native tokenization with teacher-token expansions of individual student tokens. Hybrid-prefix attention controls which prefix each expansion can access, and the last hidden state in each expansion supplies a student-vocabulary prediction. The adapted teacher then supervises student rollouts with forward KL. Experiments use Qwen3-4B as the teacher and Llama-3.2-1B and OLMo-2-1B as students, with evaluation on six math and coding benchmarks.

Critical strengths
  1. The method offers a useful way to adapt a teacher to a student's output space while retaining the teacher's native embeddings and backbone. PTS and HPA address a real difficulty that simply replacing the output head would leave unresolved: obtaining teacher states at student-token boundaries without feeding the entire teacher only independently retokenized fragments. The construction is concrete, and the ablations support the value of preserving the native teacher prefix and training the new head.

  2. The empirical gains are consistent across the two student families. In Table 1, DVLM improves the six-task average over BPM from 21.37 to 23.53 for Llama and from 22.83 to 24.72 for OLMo, and wins on each individual task against the reported cross-tokenizer baselines. I also appreciate the supplementary teacher evaluations: Table 9 shows largely preserved math and coding performance, and the bits-per-byte comparison in A.6.3 is more meaningful than comparing raw token losses across vocabularies. Together with the reported training cost, these results make a reasonable case for practical usefulness.

Critical weaknesses
  1. The scope of the claimed alignment needs to be stated more carefully. A.3 discards samples containing student tokens that cannot be decoded independently, including partial multibyte characters. This is a substantive restriction for a method motivated by complete token and vocabulary alignment. The paper does not quantify the discarded fraction or explain clearly what happens when fresh on-policy rollouts contain such tokens. A full-sized output vector alone does not establish that arbitrary student-token histories are supported. This matters for both the generality of the method and the distribution of rollouts actually used for training.

  2. The experiments establish that the full pipeline is useful, but leave some uncertainty about what accounts for its advantage. DVLM additionally trains a large new head on approximately 0.22B tokens; matching the Stage II settings does not match this extra supervision for the other methods. The component ablations help, but the no-HPA variant uses ordinary causal attention on the concatenated sequence, which lets the expansion stream access future native-teacher tokens. Its poor downstream performance therefore does not isolate the benefit of this particular HPA construction over another valid causal alternative. I would like a clearer separation between the benefits of the additional head adaptation and the proposed context construction. These concerns limit the strength of the mechanistic claims, although they do not erase the consistent end-to-end gains.

Questions for authors
  1. What fraction of Stage I examples and Stage II rollouts is affected by the independently undecodable-token restriction in A.3, for each student? Are affected rollouts discarded, repaired, or scored in another way? Please clarify whether all student vocabulary entries remain available during generation.

  2. Can you clarify the exact configuration used for Table 1? In the linked repository, DVLM-based OPD.py sets changed_teacher_tokenizer to Qwen3-4B even after loading a student-vocabulary output head. The trainer constructs its output-vocabulary mapping from this setting and calls the hybrid matched/unmatched loss. With the student tokenizer and beta=0, this can reduce to full-vocabulary forward KL, so the hybrid implementation is not by itself a contradiction. However, the published tokenizer setting appears inconsistent with the head's row semantics. Was this setting changed for the reported experiments, and can the corresponding runnable configuration be provided?

  3. Do the HPA logits computed from a complete rollout agree with logits computed incrementally from each student prefix? The native teacher tokenization and alignment groups are constructed from the complete text, whereas A.6.3 describes retokenizing the growing prefix during generation. A prefix-consistency check around token-boundary changes would help establish the connection between these procedures. I am asking for verification of equivalence, rather than asserting that future-token leakage occurs.

  4. Is there a control that gives a competing method access to the Stage I corpus or a comparable adaptation budget? If not, please narrow the attribution of the gain to the full DVLM pipeline. Also, please clarify the meaning of n=16 and n=8 at temperature 0 in Table 6, and define precisely the variation reported across the three runs.

1 comment
AuthorsOpenReview ↗

Rebuttal to Reviewer Ejwy

We sincerely thank you for your valuable comments to help improve our paper.

For W1&&Q1 'student tokens that cannot be decoded independently':

  • In the Stage I training corpus, fewer than 0.1% of the samples exhibit this issue, making it a rare occurrence.
  • We monitor this issue in our prior code and do not observe it during the OPD process. We further conduct a statistical analysis on a multilingual corpus [1], and find that this issue occurs in approximately 0.3% of all the multilingual tokens.
  • Our case study shows that this phenomenon mainly occurs with division symbols and special tokens such as emoticons and symbols. Results in Table 1 indicate that removing these samples does not affect the overall results.

For W2 'some uncertainty about what accounts for its advantage':

  • We continue training the original teacher on the same additional data and rerun the GOLD baseline with the resulting checkpoint. The table below shows that the extra data helps DVLM’s new LM head converge. The original teacher’s mapping to its native vocabulary had already converged, so the extra training has little effect on baselines, such as GOLD.
GSM8KMATH500AMC23HumanEval+CRUXEvalLCBench
GOLD43.04(0.2)27.27(0.4)5.93(0.4)34.35(0.9)13.94(0.2)1.48(0.1)
GOLD+extra43.09(0.3)26.93(0.4)5.62(0.4)35.36(0.9)13.65(0.2)1.55(0.1)
DVLM(Llama)45.89(0.2)29.93(0.3)9.06(0.3)36.99(0.9)16.44(0.4)2.86(0.1)
  • Table 7 reports the training times for both stages. With the GPU count held fixed, Stage I increases the training cost by only about 8% in GPU-hours for both teacher–student pairs. The ablations in Table 2, including 'w/o LM Head', show the benefits of constructing the context with PTS and HPA. Together, these components enable effective training of the new LM head, which in turn improves Stage II OPD.
  • In the HPA ablation, expansion tokens can indeed attend to native teacher tokens. We therefore introduce Prefix-Aligned Causal Attention (PACA), a baseline that restricts cross-stream attention to native teacher tokens within the text prefix preceding each expansion token, thereby preventing future information leakage. The results below show that HPA outperforms PACA.
GSM8KMATH500AMC23HumanEval+CRUXEvalLCBench
DVLM(Llama)+PACA44.1525.133.6433.5414.081.52
DVLM(Llama)+HPA45.8929.939.0636.9916.442.86
DVLM(Llama)-HPA38.3618.402.5823.1712.431.02

For Q2 'clarify the exact configuration':

  • Thank you for reviewing our repository. In our OPD runs, changed_teacher_tokenizer in DVLM-based OPD.py points to the new vocabulary file produced in Stage I. Qwen3-4B serves as a default to facilitate experimental analysis across different configurations, and the training launch script overrides it. We will release the final code and provide detailed training launch arguments in the non-anonymous GitHub repository.

For Q3 'a prefix-consistency check':

  • The two serve different purposes. In all OPD runs, we follow the standard paradigm: first generate a complete student rollout and then feed its full text into the models. In Stage II, the student receives the complete rollout, while the teacher receives the PTS constructed from that rollout. Although the distribution induced by PTS under HPA is not guaranteed to agree exactly with that of the original rollout, it maintains training-inference consistency with Stage I. The method therefore remains effective.
  • Appendix A.6.3 describes how we evaluate the reasoning capabilities of the DVLM teacher itself. Because its input and output vocabularies differ, we implement an approximate token-by-token generation procedure to handle its mismatch in existing evaluation frameworks.

For Q4 'a comparable adaptation budget' and 'clarify some details':

  • The first table in this rebuttal reports distillation results for a baseline that uses the Stage I corpus. DVLM outperforms this baseline under a comparable additional training budget.
  • In Table 6, n has the same meaning as in the pass@k formula: the number of responses generated per question. As described in lines 1070–1071, the variation is the larger of the absolute values of the upper and lower variance bounds, rounded to one decimal place.

References

[1] HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models

Reviewer_SCYROpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses cross-tokenizer on-policy distillation, where the teacher and student split text into different tokens (input misalignment) and predict over different vocabularies (output misalignment). Instead of approximating an alignment, it makes the teacher speak the student’s vocabulary.

  • DVLM teacher. A Dual-Vocabulary Language Model keeps the teacher backbone frozen and trains a new LM head that outputs logits over the student vocabulary.
  • Parallel-Tokenized Sequence (PTS). The teacher’s input concatenates its native tokenization with each student token re-tokenized into teacher tokens.
  • Hybrid-Prefix Attention (HPA). This attention mask lets each re-tokenized group see only the native teacher prefix up to its alignment group, avoiding a train–inference mismatch. The last hidden state of each group yields one full-dimensional teacher distribution per student token.
  • Distillation. The student is then distilled with plain forward KL.

With Qwen3-4B as the teacher and Llama-3.2-1B and OLMo-2-1B as students, DVLM beats SFT, GRPO and four cross-tokenizer baselines (DSKD, ALM, GOLD, BPM) on six math and code benchmarks.

Critical strengths
  1. A clean idea that removes the problem rather than approximating around it. Other methods aggregate spans, marginalize to bytes, or keep only shared logit dimensions. DVLM gives a token-for-token, full-vocabulary teacher distribution, so standard KL applies directly.

  2. It is lightweight. Only the new head is trained: 0.26–0.33B parameters, about 0.08–0.09 single-GPU days, roughly 8% extra cost. OPD itself runs faster than GOLD (0.23 vs. 0.35 days for Llama, App. A.6.4).

  3. Teacher quality is preserved.

  • Bits per byte on held-out text match the original teacher (Table 8).
  • Understanding benchmarks stay within about ±1 point (Fig. 4b).
  • MATH-500 and HumanEval+ drop by at most 1.6 points (Table 9).
  1. The gains are consistent. DVLM is best on both students and on average beats the strongest baseline (BPM) by 2.16 and 1.89 points. Results are averaged over 3 seeds with variance reported. The largest gains come on the hardest benchmarks: AMC23 rises from 4.68 to 9.06 for Llama. The method also wins in-domain on Countdown and LeetCode (Fig. 3).

  2. The ablations and extra comparisons are thorough. Every component matters (Table 2). The appendix also compares against tokenizer transfer (TokAlign) and a same-tokenizer teacher, and analyzes training-data difficulty. Code and data are released.

Critical weaknesses
  1. A closely related cross-tokenizer method is missing. “Knowledge Distillation for Large Language Models through Residual Learning” (ICLR 2026, openreview.net/forum?id=Dh6KxUxG20) handles cross-tokenizer distillation by aligning tokens with cross-model attention over low-dimensional teacher representations. It reports outperforming DSKD and ALM in cross-tokenizer setups such as Mixtral-8×7B → TinyLLaMA-1.1B and GPT2-120M. It belongs in the representation-space category (II) of App. A.1 alongside DSKD. Please discuss it; a comparison would be a bonus.

  2. Several ablations fall well below the untrained student. Examples: Llama “w/o Re-token.” scores 14.45 against 20.43 for the base student, and OLMo “w/o HPA” scores 9.80 on MATH-500 against 21.00. Reporting the Stage-I loss or BPB of these ablated heads would show whether the drop comes from a poorly trained head or from the misalignment itself.

  3. Two baseline numbers should agree but don’t. “w/o LM Head” is described as GOLD with the original teacher, yet it scores 19.95 and 20.53, against 21.00 and 21.83 for GOLD in Table 1. Please clarify how the two settings differ.

  4. The handling of undecodable tokens is unclear. Samples with student tokens that cannot be decoded independently (partial multibyte characters) are discarded during preprocessing (A.3).

  • What fraction of samples does this remove?
  • What happens when the student generates such a token during OPD? This matters more for non-English text.
  1. A minor inconsistency in the data filtering. The text says DAPO pass rates come from teacher rollouts, but the dataset linked in the footnote appears to use Qwen3-4B-Base.
Questions for authors
  1. Why forward KL rather than the reverse KL more common in OPD? Was reverse KL tried?

  2. Can the same backbone carry several student-vocabulary heads at once, so one DVLM teacher serves multiple students?

3 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer SCYR

We sincerely appreciate your detailed and specific feedback on the paper.

For W1 'a closely related method':

  • Thank you for your suggestion. We reproduce the method in this paper under a fair experimental setting. The results are presented in the table below, demonstrating that our DVLM outperforms this method.
GSM8KMATH-500AMC23HumanEval+CRUXEvalLCBench
paper44.63(0.4)25.67(0.4)7.39(0.3)35.57(0.9)15.92(0.2)1.85(0.1)
DVLM(Llama)45.89(0.2)29.93(0.3)9.06(0.6)36.99(0.9)16.44(0.2)2.86(0.1)

For W2 'ablations fall below untrained students':

  • Thank you for your suggestion. We report the loss and BPB values for these two ablation settings in the table below. The results demonstrate that neither of these two ablation settings enables the new teacher to fully align with or converge to the level of the original teacher.
  • HPA enables the DVLM teacher to leverage the original teacher prefix as much as possible, thereby maintaining consistency between training and inference. The retokenized sequence facilitates the training of student tokens that do not exist in the teacher vocabulary and is a crucial component for the convergence of the new LM head.
lossBPB
DVLM(Llama)1.930.37
w/o HPA1.970.39
w/o Re-token.2.010.42

For W3 'two numbers should agree but don’t':

  • In GOLD, the teacher model takes the student's rollout as input, whereas in the "w/o LM Head" setting, the teacher model takes the PTS constructed from the student's rollout as input. This difference accounts for the numerical discrepancies, and we will clarify this point in the revision.

For W4 'undecodable tokens':

  • In the Stage I training corpus, fewer than 0.1% of the samples exhibit this issue, making it a rare occurrence.
  • We monitor this issue in our prior code and do not observe it during the OPD process. We further conduct a statistical analysis on a multilingual corpus [1], and find that this issue occurs in approximately 0.3% of all the multilingual tokens.
  • Our case study shows that this phenomenon mainly occurs with division symbols and special tokens such as emoticons and symbols. Results in Table 1 indicate that removing these samples does not affect the overall results.

For W5 'inconsistency in the data filtering':

  • Our teacher model is just the Qwen3-4B-Base.

For Q1 'why forward KL rather than reverse KL':

  • Recent work suggests that the choice between FKL and RKL depends on the distillation settings. Some studies show that FKL can outperform RKL [2][3]. As shown in the table below, FKL yields better OPD performance than RKL in our preliminary experiments. We therefore adopt FKL as our distillation objective.
GSM8KMATH500AMC23HumanEval+CRUXEvalLCBench
DVLM(Llama)+RKL43.27(0.2)28.87(0.3)7.55(0.4)35.97(0.9)15.62(0.3)2.35(0.2)
DVLM(Llama)+FKL45.89(0.2)29.93(0.3)9.06(0.6)36.99(0.9)16.44(0.2)2.86(0.1)

For Q2 'carry several student-vocabulary heads at once':

  • This approach is feasible in principle and could draw on prior work in multilingual works. Given the differences among the student vocabularies, the new LM head may need a large and sufficiently diverse pretraining corpus to learn to distinguish tokens in the unified vocabulary and fully converge. Implementing this idea would be technically challenging and represents a promising direction for future research. Thank you for the suggestion.

References

[1] HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models

[2] ON-POLICY DISTILLATION OF LANGUAGE MODELS: LEARNING FROM SELF-GENERATED MISTAKES

[3] On-Policy Self-Distillation for Large Language Models

Reviewer_SCYROpenReview ↗

Changed to Accept

Dear Authors,

The reviewer appreciate the comprehensive consideration into related work and comparing DVLM with the SOTA methods. Given the new results, the reviewer don't have more comments. One concern is the related work should be properly handle with proper citation and all the baseline method(s), including the new results, should be added to the paper for completeness.

The reviewer happily promote this paper to Clear Accept. There is one last concern about the spectrum of model sizes being experimented. However, the reviewer understand the hardware constraints and appreciate the technical contributions of this work. This is just one thing to keep in mind, no need to run more experiments.

Good luck with the final results.

AuthorsOpenReview ↗

Thanks to Reviewer SCYR

Thank you for your positive assessment of our paper and rebuttal. In the revision, we will complete the related-work citations and add all the new results presented in the rebuttal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.