- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes GenDisKD, which combines an XLM-R-Large discriminative teacher with a BLOOM-7B1 teacher trained for label-conditioned span reconstruction. The union of their predicted labels supervises a target-language student: teacher agreement provides single-label supervision, while disagreement leads to maximizing the total probability assigned to the candidate label set. Only the discriminative student is deployed at inference time.
Critical strengths
S1: Keeping the generative teacher's computation offline and deploying only the student is a practically useful design. Figures 2 and 4 also attempt to motivate the two types of teachers through their complementary errors, rather than just reporting final scores.
S2: The experiments cover three benchmarks and 13 target languages. The appendix reports results from three runs against a re-implemented MARAL baseline under the same experimental environment, which is more convincing than comparing only against individual scores from previous papers.
Critical weaknesses
W1: The objective in Equation (10), -log(sum_{y in A} p_s(y|x)), has the same form as existing candidate-set probability losses in partial-label learning, such as Section 2.2, Equation (2) of Seo and Huh's On the Power of Deep but Naive Partial Label Learning. I therefore do not think the loss itself can simply be presented as a new contribution. I would like the authors to explain what candidate-set construction or learning mechanism this work adds beyond existing objectives, and position the contribution accordingly.
W2: The main text describes teacher scoring and student training over all candidate spans, but Appendix A.3 states that the generative teacher only processes spans selected by the discriminative teacher, without specifying the filtering rule. If a true entity boundary is filtered out, the generative teacher cannot directly provide a corrective signal for that boundary. This leaves me unsure which errors the method can actually correct. I would like the authors to specify the filtering rule and the supervision used for discarded spans, and report the coverage of true entity boundaries after filtering.
W3: The performance drop when removing the generative teacher in Table 3 shows that adding this teacher helps, but does not establish whether these gains require a generative teacher. Figure 4 compares the true entities recovered by the two types of auxiliary teachers, but does not compare the students they ultimately train. I would like to see a direct comparison of final entity-level F1 between “discriminative + generative” and “discriminative + discriminative,” with candidate spans and student training settings held fixed, along with clarification of whether the auxiliary teacher configurations are comparable. This would test the paper's central design more directly.
W4: Figure 5 mainly reports entropy, gold-label probability, and accuracy on examples where the teachers disagree, without reporting entity-level F1 on the full test set for the different losses. This leaves me unable to determine whether improvements in these subset metrics translate into gains on the final task. I would like the authors to report full-test-set F1 for the candidate-set, one-hot, and soft-label losses while keeping teacher outputs and the remaining training settings fixed, and specify how the two baselines construct their supervision targets when the teachers disagree.
Questions for authors
Q1: Is the student initialized from the original XLM-R-Large checkpoint, or does it inherit the parameters of the discriminative teacher already trained on the source language?
Q2: Which validation set is used to select checkpoints and hyperparameters? Are target-language gold labels involved in model selection? This matters for whether the setting of using only source-language annotations also applies to validation.
Q3: MARAL's average scores on CoNLL/WikiAnn are 84.11/76.28 in Table 1, whereas the re-implemented results in Table 9 are 83.87/75.69. Could the authors identify the source of each baseline's scores in the main table and explain the configuration differences between the original MARAL results and the current re-implementation?
Q4: Section 5.1 specifies XLM-R-Large, but Table 4 lists the student as 0.35B. Which checkpoint was actually used, and does the parameter count include all parameters needed for deployment, including word embeddings? Is the claim of a 20-fold reduction in model size correct?
Q5: The values 76.59/82.80/82.13 correspond to de/es/nl in Table 3, but the same values correspond to Spanish/Dutch/German in Table 8. Do these two tables use the same discriminative teacher?
3 comments
Weakness#4: Thank you for the constructive suggestion. The table below reports full-test-set entity-level F1 for all six languages. We use the same teacher outputs, candidate spans, and student training settings for all three losses. When the teachers disagree, both baselines select the teacher that assigns the higher probability to its own top-ranked label. The one-hot baseline uses that label as a hard target, while the soft-label baseline uses the selected teacher’s full label distribution.
One-hot supervision outperforms soft-label supervision in all three CoNLL languages and achieves the highest F1 in Dutch. On WikiAnn, soft-label supervision outperforms one-hot supervision in all three languages, and candidate-set supervision achieves the highest F1 in each. Overall, the candidate-set loss yields the highest average F1 on both datasets. These results are consistent with our motivation for retaining both predictions: when the higher-confidence label is incorrect, the other teacher’s label remains available as a candidate. We will add the full-test results and clarify how the baseline targets are constructed in the revised manuscript.
| Training loss | de | es | nl | avg. | ar | hi | zh | avg. |
|---|---|---|---|---|---|---|---|---|
| Candidate-set (GenDisKD) | 81.70 | 86.88 | 86.03 | 84.87 | 77.14 | 85.08 | 66.88 | 76.37 |
| One-hot | 81.11 | 86.13 | 86.30 | 84.51 | 69.89 | 81.36 | 61.23 | 70.83 |
| Soft-label | 79.98 | 85.85 | 84.54 | 83.46 | 74.81 | 84.34 | 64.78 | 74.64 |
Question#1: The student is initialized from the original XLM-R-Large checkpoint. We will clarify this in the implementation details. All student comparisons use the same initialization protocol.
Question#2: We select checkpoints and hyperparameters using English validation set based on F1. We will explicitly document validation usage to accurately characterize the experimental setting.
Question#3: MARAL’s scores of 84.11/76.28 in Table 1 are taken from the original paper, whereas 83.87/75.69 in Table 9 are averages over three runs of our reproduction using the public code. The verified configuration differences are GPUs.
Question#4: Thank you for noting this discrepancy. The 0.35B figure was taken from the legacy Transformers v2.4.0 documentation, which incorrectly lists XLM-R-Large as approximately 355M parameters. The full backbone contains approximately 560M parameters, including embeddings. We will correct Table 4 and revise the parameter-reduction ratio accordingly.
Question#5: Thank you for catching this discrepancy. Both tables use the same discriminative teacher. The column labels in Table 3 are correct, whereas those in Table 8 were inadvertently misordered. The values 76.59/82.80/82.13 correspond to German/Spanish/Dutch, respectively. We apologize for this labeling error and will correct Table 8 accordingly.
Dear Reviewer fMjR,
We are reaching out to follow up on our response to your concerns. We have thoroughly considered all your comments and provided the necessary clarifications. We highly value your expertise and would like to know if you have any further feedback or concerns. We are dedicated to addressing any outstanding matters you may have raised before the discussion period ends.
Best regards,
The Authors of Paper 37769
We sincerely thank you for your thoughtful comments and for recognizing GenDisKD holding practically useful design and convincing experimental results. We would like to address your concerns in detail below.
Weakness#1: Thank you for pointing out this connection. We agree that Eq. (10) has the same form as the previous partial-label learning studies. We will revise the contribution statement to read “In heterogeneous multi-teacher distillation, we construct candidate label sets from disagreeing teacher predictions and use them as partial-label supervision. This retains alternative predictions and reduces the risk of propagating errors from any one teacher.”
Weakness#2: Regarding the ambiguity of span filtering rules, we clarify that the generative teacher scores all labels only for spans the discriminative teacher predicts as entities. Its corrections are therefore limited to retained boundaries. On CoNLL, the retained candidates yield an average oracle span F1 of 91.13%, compared with 80.51% for the discriminative teacher’s hard predictions (Table 8). Using the same discriminative anchor, generative predictions recover more gold entities missed by the anchor than additional discriminative predictions in five of six languages (Figure 4).
Weakness#3: Thank you for suggesting this comparison. We trained students with either a generative teacher or a second discriminative teacher as the auxiliary teacher. Across three student seeds, Discriminative + Generative achieves 84.87 ± 0.08 entity-level F1 on CoNLL and 76.37 ± 0.16 on WikiAnn, compared with 83.49 ± 0.05 and 71.33 ± 0.16 for Discriminative + Discriminative. The gains are 1.38 and 5.04 F1 points, respectively, with higher mean F1 in all six languages. We will add these results and the teacher configurations to the revised manuscript.
Further, with the same primary discriminative teacher, pairing it with a generative teacher yields a higher oracle union F1 in all six languages (76.5 vs. 69.7 on average). The generative teacher achieves a lower standalone average F1 than the second discriminative teacher (63.9 vs. 66.2), yet its predictions yield a higher oracle ceiling when combined with the primary teacher. This suggests that the generative teacher contributes more complementary correct predictions.