- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a data selection framework for multilingual extraction tasks that jointly considers translation quality and example difficulty to improve cross-lingual transfer. The authors introduce a deterministic scoring function that combines machine translation metrics with LLM-based label transfer accuracy and loss-based difficulty to filter translated training instances. Their validation across diverse extraction tasks and 20 languages demonstrates that this composite selection strategy achieves superior performance and data efficiency compared to random or quality-only baselines.
Critical strengths
-
The proposed framework is remarkably simple and straightforward to implement, as it leverages readily available quality and difficulty metrics without requiring complex model architecture changes.
-
The empirical results consistently demonstrate that even a straightforward combination of these metrics is highly effective in enhancing the performance of multilingual extraction, providing a practical solution for cross-lingual transfer.
Critical weaknesses
-
The proposed multi-dimensional filtering approach closely mirrors existing data selection methods that have become standard in instruction tuning. For instance, recent works on D3 (Diversity, Difficulty, and Dependability) [1] and on systematic data selection [2] demonstrate that such composite criteria can be helpful. Given that many of these components (e.g., MT metrics, LLM-based judges, and loss-based selection) are already considered standard components in contemporary data-centric pipelines, the current study's contribution seems to be a straightforward application of these general techniques to multilingual extraction, rather than a fundamental methodological innovation.
-
Some empirical results and interpretation are consistent with standard expectations, where training with translated training dataset leads to better performance (Figure 3). The paper lacks a comparison with other methods and deeper analysis into why this specific aggregation method works better than existing approaches, leaving the core contribution to be perceived as an combination of general techniques rather than a meaningful advancement in multilingual extraction.
[1] D3: Diversity, Difficulty, and Dependability-Aware Data Selection for Sample-Efficient LLM Instruction Tuning
[2] What makes good data for alignment? A comprehensive study of automatic data selection in instruction tuning
Questions for authors
- Have you conducted a comparative study against other data selection or translation strategies for multilingual extraction (ex. [3])? Please clarify the core methodological distinction between your proposed approach and these established techniques, beyond the simple aggregation of quality and difficulty metrics.
[3] Contextual Label Projection for Cross-Lingual Structured Prediction
2 comments
- Comparison with other Label Projection strategies (e.g., CLaP [3]):
CLaP is a label projection method, which determines how the labels are transferred during translation, while our method selects which translated examples to use for training. The two approaches are therefore complementary, and our selection can be applied on top of any label projection method. In fact, our correlation study in Section 5 already includes CLaP. In this study, no data selection is applied: for each dataset and target language, we translate the full English training data with either mark-then-translate or CLaP, fine-tune Gemma-3-270M-IT on the English data together with the translated data, and evaluate on the test set of that language. The table below compares the two projection methods using Gemma-3-27B-IT and Qwen-3-30B-A3B-IT as the translation models. The reported results are averaged over all languages in each dataset (average F1 in %, with each model trained with three random seeds):
| Dataset (#langs) | MT=Gemma-3-27B (Marker / CLaP) | MT=Qwen-3-30B (Marker / CLaP) |
|---|---|---|
| RED^FM (6) | 11.8 / 10.2 | 12.8 / 11.7 |
| MASSIVE (5) | 34.4 / 36.4 | 32.4 / 34.1 |
| E3C (4) | 26.0 / 21.3 | 23.9 / 22.8 |
Marker denotes Mark-then-translate
For both translation models, neither projection method is consistently better: mark-then-translate performs better on RED^FM and E3C, while CLaP performs better on MASSIVE.
[3] Contextual Label Projection for Cross-Lingual Structured Prediction
- Core methodological distinction
We agree that quality- and difficulty-based data selection have a rich history; however, how these criteria should be designed and applied can differ across settings and downstream tasks, and this remains understudied for multilingual extraction. In this work, we investigate how these criteria should be applied in this setting.
Firstly, while components such as XCOMET are widely used to assess translation quality, it remains unclear whether they are sufficient for assessing the quality of translated training data for extraction tasks. In Section 5, we systematically study this question, and explain when XCOMET is sufficient and when it benefits from being complemented by an LLM judge.
Secondly, how to use the difficulty signal to select the data also needs to be studied in this setting. As shown in the previous point, directly applying D3, a selection method designed for open-ended instruction tuning, does not transfer well to extraction tasks and leads to oversampling of examples without labels. The reviewer’s suggested comparison with D3 has helped strengthen our analysis, providing further evidence that targeting a specific difficulty range is more suitable for extraction tasks. We have added this analysis to Appendix B.5.
Thank you for your insightful feedback. Please find below our response to each point of your comments and some updates in the paper based on your suggestions.
Update on experimental results
While implementing the D3 baseline suggested by the reviewer, we identified an implementation issue in our difficulty-scoring pipeline: data shuffling in the dataloader caused the difficulty scores to be misaligned with the example IDs. This issue only affects the selection methods that use difficulty scores (e.g., Quality-Difficulty), while Random, Quality, and the correlation study in Section 5 are unaffected. We have fixed the issue and re-run all affected experiments with the same hyperparameters, and the main experiments in Figure 2 are now run with three random seeds. We have updated the paper accordingly. With the corrected scores, our main conclusion holds: Quality-Difficulty outperforms Quality on all 8 datasets, and is significantly better in 51 of the 72 task–language pairs. One change is that Difficulty-only now performs substantially below Quality. As we investigate in Appendix B.5, this is partly because prioritizing the highest-loss examples leads to over-selecting examples without labels, specifically in tasks where these examples are rare. This is consistent with our motivation for prioritizing examples within a target difficulty range rather than the most difficult ones.
We have implemented D3, a data selection method that jointly considers diversity, difficulty, and dependability. For a fair comparison, D3 uses the same English-only model as the seed model and Gemma-3-27B-IT as the teacher model, and follows the same per task–language selection setup. We compare the following selection methods: Quality, Quality-Difficulty, D3, and Difficulty-only (a simple baseline that prioritizes the examples with the highest loss under the English-only model). The table below reports the average performance (in %) over 8 test datasets of these selection methods at the 1× translation ratio. All experiments are run with 3 random seeds.
From the table, we see that Difficulty-only and D3 achieve substantially lower average performance than Quality-Difficulty. One reason is that both methods prioritize the examples with high difficulty, which leads to over-selecting training examples with no label.For example, in BC5CDR, an English dataset included in our training data, only 16% of the examples have no labels. However, among the examples selected by D3 from the translated version of this dataset, 76% contain no labels. This substantial distribution shift leads the model to overpredict empty outputs: D3's model produces no labels for 25% of test instances that contain labels, compared with only 12% for the Quality-Difficulty method. This phenomenon can be attributed to two factors. First, in extraction tasks, the expected output of an instance with no label is an empty list, which means these instances always have perfect label transfer accuracy. Second, difficulty is defined as the mean loss per token. When the seed model incorrectly predicts label spans for an instance with no labels, the resulting loss per token tends to be high because the output contains only a few tokens. In contrast, errors on other instances are averaged over a larger number of tokens, resulting in lower mean loss per token.
We hypothesize that D3 does not work well in the extraction task setting because it is designed for open-ended instruction-following tasks, where an instruction may have many acceptable responses. According to the D3 paper, a high-loss example can either be truly difficult or simply reflect another valid response that the model does not prefer, so D3 uses the model’s uncertainty (entropy) to avoid treating the latter case as difficult. In extraction tasks, however, the output space is much more limited, so this design no longer filters out unwanted cases, such as no-label examples. Unlike D3, which ranks examples higher as their difficulty increases, our method down-weights extremely difficult examples, which may be outliers.