- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a method for multilingual on-policy self-distillation that selects a language to encode privileged information for the teacher separately for each training example. During distillation, the student receives each question in the target language, while the teacher has access to privileged information (e.g., the answer) in a language that is selected according to the proposed method. The privileged information language is selected by measuring which candidate context minimizes the perplexity of the target-language reference answer under the model. Overall, I think this is a promising approach.
Critical strengths
I am generally positive about the paper. The experiments show consistent improvements, with encouraging results on CARE. The comparisons against fixed-English, random-language, and target-language baselines show that the proposed selection procedure is effective.
Critical weaknesses
My main concern is the lack of a baseline using the source language of each dataset example as the teacher’s privileged context. As this seems like the most natural way to do OPSD in this scenario, rather than translating the input which is what the paper is proposing.
The paper shows improved performance over English and the target language, but doesn't compare to just using the original language of the question. It would also be interesting to see how often the proposed selection method chooses the original source language.
I think the paper could also do a better job of explaining why the proposed method should be expected to work better than the english-only or original language baselines. The paper offers an explanation based on making knowledge accessible through different linguistic pathways, which seems plausible, but the description of the mechanism for how this might happen is left a bit vague. It would be nice to have a more specific hypothesis for how this works, along with some evidence to support it.
Questions for authors
see above