acceptodds
Under review as a conference paper at ICLR 2027

Generative-Discriminative Knowledge Distillation for Cross-Lingual NER

Abstract

Cross-lingual named entity recognition (NER) aims to build an NER model that generalizes to the low-resource target languages with labeled data from the high-resource source language. Existing approaches can be broadly categorized into two paradigms: discriminative methods based on small language models and generative methods based on large language models. These paradigms are complementary: discriminative models enable fast and efficient inference but may overfit source-language surface features, whereas generative models are generally more robust to unseen languages but incur substantial computational costs during decoding. To combine their complementary strengths, we propose a **Gen**erative-**Dis**criminative **K**nowledge **D**istillation **(GenDisKD)** framework, which transfers knowledge from both discriminative and generative teachers to a compact student model for cross-lingual NER. Furthermore, we introduce a candidate-set distillation loss that preserves alternative candidate labels when teachers disagree, allowing the student to resolve ambiguities through complementary supervision. Extensive experiments on three benchmarks demonstrate that GenDisKD consistently outperforms existing discriminative and generative methods, validating its effectiveness and robustness.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
cross-lingual, named entity recognition, knowledge distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_fMjROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GenDisKD, which combines an XLM-R-Large discriminative teacher with a BLOOM-7B1 teacher trained for label-conditioned span reconstruction. The union of their predicted labels supervises a target-language student: teacher agreement provides single-label supervision, while disagreement leads to maximizing the total probability assigned to the candidate label set. Only the discriminative student is deployed at inference time.

Critical strengths

S1: Keeping the generative teacher's computation offline and deploying only the student is a practically useful design. Figures 2 and 4 also attempt to motivate the two types of teachers through their complementary errors, rather than just reporting final scores.

S2: The experiments cover three benchmarks and 13 target languages. The appendix reports results from three runs against a re-implemented MARAL baseline under the same experimental environment, which is more convincing than comparing only against individual scores from previous papers.

Critical weaknesses

W1: The objective in Equation (10), -log(sum_{y in A} p_s(y|x)), has the same form as existing candidate-set probability losses in partial-label learning, such as Section 2.2, Equation (2) of Seo and Huh's On the Power of Deep but Naive Partial Label Learning. I therefore do not think the loss itself can simply be presented as a new contribution. I would like the authors to explain what candidate-set construction or learning mechanism this work adds beyond existing objectives, and position the contribution accordingly.

W2: The main text describes teacher scoring and student training over all candidate spans, but Appendix A.3 states that the generative teacher only processes spans selected by the discriminative teacher, without specifying the filtering rule. If a true entity boundary is filtered out, the generative teacher cannot directly provide a corrective signal for that boundary. This leaves me unsure which errors the method can actually correct. I would like the authors to specify the filtering rule and the supervision used for discarded spans, and report the coverage of true entity boundaries after filtering.

W3: The performance drop when removing the generative teacher in Table 3 shows that adding this teacher helps, but does not establish whether these gains require a generative teacher. Figure 4 compares the true entities recovered by the two types of auxiliary teachers, but does not compare the students they ultimately train. I would like to see a direct comparison of final entity-level F1 between “discriminative + generative” and “discriminative + discriminative,” with candidate spans and student training settings held fixed, along with clarification of whether the auxiliary teacher configurations are comparable. This would test the paper's central design more directly.

W4: Figure 5 mainly reports entropy, gold-label probability, and accuracy on examples where the teachers disagree, without reporting entity-level F1 on the full test set for the different losses. This leaves me unable to determine whether improvements in these subset metrics translate into gains on the final task. I would like the authors to report full-test-set F1 for the candidate-set, one-hot, and soft-label losses while keeping teacher outputs and the remaining training settings fixed, and specify how the two baselines construct their supervision targets when the teachers disagree.

Questions for authors

Q1: Is the student initialized from the original XLM-R-Large checkpoint, or does it inherit the parameters of the discriminative teacher already trained on the source language?

Q2: Which validation set is used to select checkpoints and hyperparameters? Are target-language gold labels involved in model selection? This matters for whether the setting of using only source-language annotations also applies to validation.

Q3: MARAL's average scores on CoNLL/WikiAnn are 84.11/76.28 in Table 1, whereas the re-implemented results in Table 9 are 83.87/75.69. Could the authors identify the source of each baseline's scores in the main table and explain the configuration differences between the original MARAL results and the current re-implementation?

Q4: Section 5.1 specifies XLM-R-Large, but Table 4 lists the student as 0.35B. Which checkpoint was actually used, and does the parameter count include all parameters needed for deployment, including word embeddings? Is the claim of a 20-fold reduction in model size correct?

Q5: The values 76.59/82.80/82.13 correspond to de/es/nl in Table 3, but the same values correspond to Spanish/Dutch/German in Table 8. Do these two tables use the same discriminative teacher?

3 comments

We sincerely thank you for your thoughtful comments and for recognizing GenDisKD holding practically useful design and convincing experimental results. We would like to address your concerns in detail below.

Weakness#1: Thank you for pointing out this connection. We agree that Eq. (10) has the same form as the previous partial-label learning studies. We will revise the contribution statement to read “In heterogeneous multi-teacher distillation, we construct candidate label sets from disagreeing teacher predictions and use them as partial-label supervision. This retains alternative predictions and reduces the risk of propagating errors from any one teacher.”

Weakness#2: Regarding the ambiguity of span filtering rules, we clarify that the generative teacher scores all labels only for spans the discriminative teacher predicts as entities. Its corrections are therefore limited to retained boundaries. On CoNLL, the retained candidates yield an average oracle span F1 of 91.13%, compared with 80.51% for the discriminative teacher’s hard predictions (Table 8). Using the same discriminative anchor, generative predictions recover more gold entities missed by the anchor than additional discriminative predictions in five of six languages (Figure 4).

Weakness#3: Thank you for suggesting this comparison. We trained students with either a generative teacher or a second discriminative teacher as the auxiliary teacher. Across three student seeds, Discriminative + Generative achieves 84.87 ± 0.08 entity-level F1 on CoNLL and 76.37 ± 0.16 on WikiAnn, compared with 83.49 ± 0.05 and 71.33 ± 0.16 for Discriminative + Discriminative. The gains are 1.38 and 5.04 F1 points, respectively, with higher mean F1 in all six languages. We will add these results and the teacher configurations to the revised manuscript.

Teacher configurationdeesnlavg.arhizhavg.
Discriminative + Discriminative79.30 ± 0.0485.77 ± 0.0885.39 ± 0.1483.49 ± 0.0568.80 ± 0.1282.88 ± 0.2962.32 ± 0.2871.33 ± 0.16
Discriminative + Generative81.70 ± 0.0886.88 ± 0.1786.03 ± 0.4084.87 ± 0.0877.14 ± 0.4185.08 ± 0.2366.88 ± 0.0976.37 ± 0.16

Further, with the same primary discriminative teacher, pairing it with a generative teacher yields a higher oracle union F1 in all six languages (76.5 vs. 69.7 on average). The generative teacher achieves a lower standalone average F1 than the second discriminative teacher (63.9 vs. 66.2), yet its predictions yield a higher oracle ceiling when combined with the primary teacher. This suggests that the generative teacher contributes more complementary correct predictions.

Span-level F1 (%)arzhhiesnldeavg.
Primary discriminative teacher47.045.271.576.784.073.566.3
Auxiliary discriminative teacher46.945.969.177.784.373.266.2
Auxiliary generative teacher (Figure 2)58.841.671.176.470.564.963.9
Oracle union: Discriminative + Discriminative60.047.271.579.185.574.569.7
Oracle union: Discriminative + Generative (Figure 2)65.754.182.686.188.382.376.5

Weakness#4: Thank you for the constructive suggestion. The table below reports full-test-set entity-level F1 for all six languages. We use the same teacher outputs, candidate spans, and student training settings for all three losses. When the teachers disagree, both baselines select the teacher that assigns the higher probability to its own top-ranked label. The one-hot baseline uses that label as a hard target, while the soft-label baseline uses the selected teacher’s full label distribution.

One-hot supervision outperforms soft-label supervision in all three CoNLL languages and achieves the highest F1 in Dutch. On WikiAnn, soft-label supervision outperforms one-hot supervision in all three languages, and candidate-set supervision achieves the highest F1 in each. Overall, the candidate-set loss yields the highest average F1 on both datasets. These results are consistent with our motivation for retaining both predictions: when the higher-confidence label is incorrect, the other teacher’s label remains available as a candidate. We will add the full-test results and clarify how the baseline targets are constructed in the revised manuscript.

Training lossdeesnlavg.arhizhavg.
Candidate-set (GenDisKD)81.7086.8886.0384.8777.1485.0866.8876.37
One-hot81.1186.1386.3084.5169.8981.3661.2370.83
Soft-label79.9885.8584.5483.4674.8184.3464.7874.64

Question#1: The student is initialized from the original XLM-R-Large checkpoint. We will clarify this in the implementation details. All student comparisons use the same initialization protocol.

Question#2: We select checkpoints and hyperparameters using English validation set based on F1. We will explicitly document validation usage to accurately characterize the experimental setting.

Question#3: MARAL’s scores of 84.11/76.28 in Table 1 are taken from the original paper, whereas 83.87/75.69 in Table 9 are averages over three runs of our reproduction using the public code. The verified configuration differences are GPUs.

Question#4: Thank you for noting this discrepancy. The 0.35B figure was taken from the legacy Transformers v2.4.0 documentation, which incorrectly lists XLM-R-Large as approximately 355M parameters. The full backbone contains approximately 560M parameters, including embeddings. We will correct Table 4 and revise the parameter-reduction ratio accordingly.

Question#5: Thank you for catching this discrepancy. Both tables use the same discriminative teacher. The column labels in Table 3 are correct, whereas those in Table 8 were inadvertently misordered. The values 76.59/82.80/82.13 correspond to German/Spanish/Dutch, respectively. We apologize for this labeling error and will correct Table 8 accordingly.

Dear Reviewer fMjR,

We are reaching out to follow up on our response to your concerns. We have thoroughly considered all your comments and provided the necessary clarifications. We highly value your expertise and would like to know if you have any further feedback or concerns. We are dedicated to addressing any outstanding matters you may have raised before the discussion period ends.

Best regards,

The Authors of Paper 37769

Reviewer_FL5TOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes GenDisKD, a generative–discriminative knowledge distillation framework for cross-language named entity recognition (NER). It combines the complementary knowledge of generative and discriminative teacher models and introduces a candidate set distillation loss to address discrepancies in teacher predictions. This research addresses the challenges of knowledge transfer and inference efficiency in low-resource cross-language NER, holding significant practical value. Experimental results show that this method achieves competitive performance on three benchmark datasets and is capable of using only a compact discriminative student model during the inference stage.

Critical strengths
  1. Named entity recognition, especially in environments with limited resources, cross-lingual, and open entities, has clear research value.
  2. This paper has discovered that the knowledge of discriminative NER models and generative models is complementary.
Critical weaknesses
  1. The paper’s contribution is described as merging the predictions from generative and discriminative teachers into a set of candidate labels, which is then used to train the student model via partial-label loss. This approach is very similar to existing multi-teacher knowledge distillation and partial-label learning methods. The authors do not seem to clearly articulate the fundamental differences between candidate set distillation and existing methods, nor do they explain why this combination constitutes a new contribution. Furthermore, the experiments in this paper do not directly prove that the proposed distillation mechanism can effectively utilize this complementary information.
  2. Performance improvements over the strongest baselines are limited, and there is a lack of statistical significance analysis. Compared to MARAL, GenDisKD achieves average F1 score improvements of only 0.76, 0.09, and 0.41 percentage points on CoNLL, WikiAnn, and MultiCoNER, respectively. In particular, the improvement on WikiAnn is very limited, making it difficult to determine whether it exceeds random experimental fluctuations. Therefore, the current experimental results are insufficient to fully demonstrate that this method possesses a stable performance advantage.
Questions for authors
  1. Candidate set distillation loss shown in Eq. 10 does not appear to directly resolve ambiguities between teacher predictions. In particular, when both teachers make incorrect predictions, the true label may not be present in the candidate set at all, and there remains a risk of erroneous supervision propagation. Please explain how the student model resolves ambiguities among candidate labels rather than merely increasing the total probability mass assigned to their candidate set.
  2. In the contribution section of this paper, it is mentioned that the prediction results are merged into a set of candidate labels, and that a student model is trained using a subset of these labels. How does this differ from existing multi-teacher knowledge distillation and partial label learning methods?
2 comments

We sincerely thank you for your thoughtful comments and for recognizing GenDisKD holding significant practical value and clear research value. We would like to address your concerns in detail below.

Weakness#1 & Question#1 & Question#2: We thank the reviewer for raising the concern on contribution description. We agree that multi-teacher knowledge distillation and partial-label learning are established methods. Our contribution is the design of GenDisKD to address overfitting in discriminative cross-lingual NER. We introduce a label-conditioned generative teacher to provide complementary evidence about entity types. When the teachers disagree on a span, partial-label supervision keeps alternative labels available for student learning. This reduces the risk of treating an erroneous teacher prediction as ground truth.

Sections 5.4.1 and 5.4.2 evaluate the two parts of this design. Section 5.4.1 shows that the generative teacher provides complementary predictions. Section 5.4.2 compares distillation losses at the student level and shows the benefit of candidate-set supervision within GenDisKD. We will revise the contribution statement and discussion to make this connection explicit.

Weakness#2:
Regarding the concern of significance analysis, we clarify the following: Appendix A.5 (Table 9) reports a three-seed comparison against MARAL reimplemented from its public code under the same experimental conditions as GenDisKD. In this comparison, GenDisKD achieves higher mean F1 in all six languages, with average gains of 1.00 F1 point on CoNLL and 0.68 on WikiAnn. On WikiAnn, GenDisKD also achieves higher average F1 on each of the three seeds. Across the two datasets, the mean gain is 0.84 F1 points, and the paired test reported in Appendix A.5 finds this overall gain statistically significant (t=8.52, p<0.001t=8.52,\ p<0.001). We will cite Table 9 in the main results section to make this evidence easier to find.

Dear Reviewer FL5T,

We are reaching out to follow up on our response to your concerns. We have thoroughly considered all your comments and provided the necessary clarifications. We highly value your expertise and would like to know if you have any further feedback or concerns. We are dedicated to addressing any outstanding matters you may have raised before the discussion period ends.

Best regards,

The Authors of Paper 37769

Reviewer_W5PrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a cross-lingual named entity recognition (NER) framework called GenDisKD. This framework constructs a candidate label set by combining generative and discriminative teacher models; by retaining candidate labels from these different teachers, it reduces over-reliance on a single pseudo-label in cases where the teachers' predictions are inconsistent. Experiments conducted on several cross-lingual NER datasets demonstrate that this approach achieves competitive performance.

Critical strengths
  1. A distillation framework combining generative and discriminative teacher models is proposed, which reduces over-reliance on a single pseudo-label by constructing a set of candidate labels.
  2. Experimental results demonstrate that GenDisKD achieves competitive performance across multiple cross-lingual NER datasets.
Critical weaknesses
  1. When the predictions from the two teachers are inconsistent, the student learns both candidate labels; however, Equation (10) does not explicitly specify how to distinguish between these two labels, while Equation (11) determines the final prediction solely based on the maximum probability.
  2. Regarding the training of generative teachers, what are the specific advantages of reconstructing the original text using candidate labels, as opposed to directly generating entity labels?
  3. The authors should evaluate the model's runtime across varying sentence lengths, given that the number of candidate segments increases as the sentence length grows.
Questions for authors

See the weaknesses.

4 comments

We sincerely thank you for your thoughtful comments and for recognizing GenDisKD reducing over-reliance on a single pseudo-label. We would like to address your concerns in detail below.

Weakness#1: Regarding the confusion on Equation (10) and Equation (11), we would like to clarify that they serve different purposes: During inferencing, Eq. (11) selects the label to which the trained student assigns the highest probability. During training, when the two teachers disagree, Eq. (10) retains both predictions as candidates and encourages the student to assign probability mass to the set as a whole. This avoids treating either potentially incorrect teacher prediction as the sole training target.

Weakness#2: Regarding the specific advantages of reconstructing object, we would like to clarify as follows. Directly generating entity labels in free form requires parsing outputs that may contain invalid labels or extra text. Our Bayes-based generative NER teacher ranks predefined entity types by the likelihood they assign to the observed span. Each candidate type is scored against the same target tokens, so the scores are comparable across types and independent of variation in generated output length. Reconstruction also changes the training signal. Predicting the span token by token given its entity type provides supervision at every position in the span. This encourages the teacher to use contextual evidence when assessing whether a candidate type fits and may reduce its reliance on shortcuts such as favoring frequent labels. Section 5.4.1 shows that the generative teacher’s predictions complement those of the discriminative teacher.

Weakness#3: Thank you for highlighting the importance of runtime analysis with respect to sentence length. We conducted an additional inference-time experiment on the German, Spanish, and Dutch target datasets of CoNLL03, grouping sentences into four length intervals: 1–16, 17–32, 33–64, and 65–128 words. The table reports the mean latency per sentence in milliseconds.

Sentence length (words)GermanSpanishDutch
1–1628.4524.8023.56
17–3257.3646.5342.26
33–64123.0199.2980.89
65–128447.61214.59282.18

Latency increases with sentence length across all three languages. We will include this timing scope and the sentence-length analysis in the revised manuscript.

Reviewer_W5PrOpenReview ↗

A Remaining Concern About Candidate-Label Disambiguation

Thank you for your responses and additional experiments. However, I still have one concern regarding Weakness 1: When the two teachers disagree, both labels are retained as candidates for student training. What enables the student to distinguish between these two candidates and assign a higher probability to the more appropriate label?

Thank you for the follow-up. Because the same student is trained on both agreement and disagreement examples, it can use the associations between context and entity type learned from agreement examples to rank the candidates on disagreement examples. The loss comparison in Section 5.4.2 supports this design at the task level. The student trained with candidate-set supervision achieves superior performance than students trained with either one-hot or soft-label supervision.

Dear Reviewer W5Pr,

We are reaching out to follow up on our response to your concerns. We have thoroughly considered all your comments and provided the necessary clarifications. We highly value your expertise and would like to know if you have any further feedback or concerns. We are dedicated to addressing any outstanding matters you may have raised before the discussion period ends.

Best regards,

The Authors of Paper 37769

Reviewer_bNrpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a Generative Discriminative Knowledge Distillation (GenDisKD) framework for cross-lingual named entity recognition, which jointly transfers discriminative classification boundaries and generative entity distributional knowledge from two teachers to train a compact student model. One teacher is a discriminative NER teacher, the other teacher is a Bayes-based generative NER teacher. The paper evaluated the proposed approach on three cross-lingual NER datasets, and showed that it achieved better results than some existing approaches.

Critical strengths

The proposed knowledge distillation framework combines the advantages of both a discriminative teacher and a generative teacher to improve the student model.

Critical weaknesses

The evaluation results showed that the improvement of the proposed approach is very small compared with the strongest prior approach (MARAL). There is no statistical significance evaluation that the proposed approach is better than MARAL.

Questions for authors

In addition to accuracy performance (f1-score), how does the proposed approach compare with existing approaches in terms of training and inference cost?

2 comments

We sincerely thank you for your thoughtful comments and for recognizing the value of combining generative and discriminative supervision.

Weakness#1: Regarding the concern of significance analysis, we clarify the following: Appendix A.5 (Table 9) reports a three-seed comparison against MARAL reimplemented from its public code under the same experimental conditions as GenDisKD. In this comparison, GenDisKD achieves higher mean F1 in all six languages, with average gains of 1.00 F1 point on CoNLL and 0.68 on WikiAnn. On WikiAnn, GenDisKD also achieves higher average F1 on each of the three seeds. Across the two datasets, the mean gain is 0.84 F1 points, and the paired test reported in Appendix A.5 finds this overall gain statistically significant (t=8.52, p<0.001t=8.52,\ p<0.001). We will cite Table 9 in the main results section to make this evidence easier to find.

Question#1: Regarding computational cost, we provide the following comparison. The measured training time were 3.99 hours for the discriminative baseline, 8.34 hours for the generative baseline and GenDisKD. At inference, as shown in Table 4, GenDisKD takes 243 seconds, compared with 222 seconds for the discriminative baseline and 1,566 seconds for the generative baseline. Thus, GenDisKD takes about 2.1 times as long to train as the discriminative baseline, while adding about 9.5% to its inference time. We will add the training-time comparison alongside Table 4 in the revised manuscript.

Dear Reviewer bNrp,

We are reaching out to follow up on our response to your concerns. We have thoroughly considered all your comments and provided the necessary clarifications. We highly value your expertise and would like to know if you have any further feedback or concerns. We are dedicated to addressing any outstanding matters you may have raised before the discussion period ends.

Best regards,

The Authors of Paper 37769

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.