acceptodds
Under review as a conference paper at ICLR 2027

Quality- and Difficulty-Aware Data Selection for Multilingual Extraction

Abstract

Training models on translated data is widely used to improve multilingual capabilities, particularly for low-resource languages where data scarcity remains a longstanding challenge. However, translated data can contain noise, making effective data selection crucial. While data quality can help identify noisy instances, it does not indicate how informative an instance is for learning. We therefore propose selecting translated data by jointly considering data quality and example difficulty. We study this approach in extraction tasks, where predictions are made over words or phrases and must be precisely grounded in the input text. Such tasks are particularly valuable targets for translation, since fine-grained annotation is costly to produce in new languages. However, they also pose a particular challenge: label annotations may become invalid after translation even when sentence-level translation quality is high. We show that machine translation metrics fail to capture such errors and complement them with label transfer accuracy, assessed using an LLM-as-a-judge. We evaluate these selection strategies by fine-tuning an LLM on English extraction tasks and their translations into 20 languages. Our results show that selection based on quality alone generally outperforms random selection, while prioritizing high-quality instances within a target difficulty range further outperforms quality-based selection alone on 7 of 8 evaluation datasets. Finally, we show that fine-tuning on translated data remains beneficial even for LLMs with strong multilingual abilities, outperforming few-shot prompting on 6 of 8 datasets.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multilingual, Span-level Extraction, Data Selection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wY9BOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a data selection framework for multilingual extraction tasks that jointly considers translation quality and example difficulty to improve cross-lingual transfer. The authors introduce a deterministic scoring function that combines machine translation metrics with LLM-based label transfer accuracy and loss-based difficulty to filter translated training instances. Their validation across diverse extraction tasks and 20 languages demonstrates that this composite selection strategy achieves superior performance and data efficiency compared to random or quality-only baselines.

Critical strengths
  • The proposed framework is remarkably simple and straightforward to implement, as it leverages readily available quality and difficulty metrics without requiring complex model architecture changes.

  • The empirical results consistently demonstrate that even a straightforward combination of these metrics is highly effective in enhancing the performance of multilingual extraction, providing a practical solution for cross-lingual transfer.

Critical weaknesses
  • The proposed multi-dimensional filtering approach closely mirrors existing data selection methods that have become standard in instruction tuning. For instance, recent works on D3 (Diversity, Difficulty, and Dependability) [1] and on systematic data selection [2] demonstrate that such composite criteria can be helpful. Given that many of these components (e.g., MT metrics, LLM-based judges, and loss-based selection) are already considered standard components in contemporary data-centric pipelines, the current study's contribution seems to be a straightforward application of these general techniques to multilingual extraction, rather than a fundamental methodological innovation.

  • Some empirical results and interpretation are consistent with standard expectations, where training with translated training dataset leads to better performance (Figure 3). The paper lacks a comparison with other methods and deeper analysis into why this specific aggregation method works better than existing approaches, leaving the core contribution to be perceived as an combination of general techniques rather than a meaningful advancement in multilingual extraction.

[1] D3: Diversity, Difficulty, and Dependability-Aware Data Selection for Sample-Efficient LLM Instruction Tuning

[2] What makes good data for alignment? A comprehensive study of automatic data selection in instruction tuning

Questions for authors
  • Have you conducted a comparative study against other data selection or translation strategies for multilingual extraction (ex. [3])? Please clarify the core methodological distinction between your proposed approach and these established techniques, beyond the simple aggregation of quality and difficulty metrics.

[3] Contextual Label Projection for Cross-Lingual Structured Prediction

2 comments

Thank you for your insightful feedback. Please find below our response to each point of your comments and some updates in the paper based on your suggestions.

Update on experimental results

While implementing the D3 baseline suggested by the reviewer, we identified an implementation issue in our difficulty-scoring pipeline: data shuffling in the dataloader caused the difficulty scores to be misaligned with the example IDs. This issue only affects the selection methods that use difficulty scores (e.g., Quality-Difficulty), while Random, Quality, and the correlation study in Section 5 are unaffected. We have fixed the issue and re-run all affected experiments with the same hyperparameters, and the main experiments in Figure 2 are now run with three random seeds. We have updated the paper accordingly. With the corrected scores, our main conclusion holds: Quality-Difficulty outperforms Quality on all 8 datasets, and is significantly better in 51 of the 72 task–language pairs. One change is that Difficulty-only now performs substantially below Quality. As we investigate in Appendix B.5, this is partly because prioritizing the highest-loss examples leads to over-selecting examples without labels, specifically in tasks where these examples are rare. This is consistent with our motivation for prioritizing examples within a target difficulty range rather than the most difficult ones.

  • Comparison with D3

We have implemented D3, a data selection method that jointly considers diversity, difficulty, and dependability. For a fair comparison, D3 uses the same English-only model as the seed model and Gemma-3-27B-IT as the teacher model, and follows the same per task–language selection setup. We compare the following selection methods: Quality, Quality-Difficulty, D3, and Difficulty-only (a simple baseline that prioritizes the examples with the highest loss under the English-only model). The table below reports the average performance (in %) over 8 test datasets of these selection methods at the 1× translation ratio. All experiments are run with 3 random seeds.

SelectionAverage score
Difficulty-only27.6 ± 0.5
D326.2 ± 1.0
Quality-Difficulty40.9 ± 0.5

From the table, we see that Difficulty-only and D3 achieve substantially lower average performance than Quality-Difficulty. One reason is that both methods prioritize the examples with high difficulty, which leads to over-selecting training examples with no label.For example, in BC5CDR, an English dataset included in our training data, only 16% of the examples have no labels. However, among the examples selected by D3 from the translated version of this dataset, 76% contain no labels. This substantial distribution shift leads the model to overpredict empty outputs: D3's model produces no labels for 25% of test instances that contain labels, compared with only 12% for the Quality-Difficulty method. This phenomenon can be attributed to two factors. First, in extraction tasks, the expected output of an instance with no label is an empty list, which means these instances always have perfect label transfer accuracy. Second, difficulty is defined as the mean loss per token. When the seed model incorrectly predicts label spans for an instance with no labels, the resulting loss per token tends to be high because the output contains only a few tokens. In contrast, errors on other instances are averaged over a larger number of tokens, resulting in lower mean loss per token.

We hypothesize that D3 does not work well in the extraction task setting because it is designed for open-ended instruction-following tasks, where an instruction may have many acceptable responses. According to the D3 paper, a high-loss example can either be truly difficult or simply reflect another valid response that the model does not prefer, so D3 uses the model’s uncertainty (entropy) to avoid treating the latter case as difficult. In extraction tasks, however, the output space is much more limited, so this design no longer filters out unwanted cases, such as no-label examples. Unlike D3, which ranks examples higher as their difficulty increases, our method down-weights extremely difficult examples, which may be outliers.

  • Comparison with other Label Projection strategies (e.g., CLaP [3]):

CLaP is a label projection method, which determines how the labels are transferred during translation, while our method selects which translated examples to use for training. The two approaches are therefore complementary, and our selection can be applied on top of any label projection method. In fact, our correlation study in Section 5 already includes CLaP. In this study, no data selection is applied: for each dataset and target language, we translate the full English training data with either mark-then-translate or CLaP, fine-tune Gemma-3-270M-IT on the English data together with the translated data, and evaluate on the test set of that language. The table below compares the two projection methods using Gemma-3-27B-IT and Qwen-3-30B-A3B-IT as the translation models. The reported results are averaged over all languages in each dataset (average F1 in %, with each model trained with three random seeds):

Dataset (#langs)MT=Gemma-3-27B (Marker / CLaP)MT=Qwen-3-30B (Marker / CLaP)
RED^FM (6)11.8 / 10.212.8 / 11.7
MASSIVE (5)34.4 / 36.432.4 / 34.1
E3C (4)26.0 / 21.323.9 / 22.8

Marker denotes Mark-then-translate

For both translation models, neither projection method is consistently better: mark-then-translate performs better on RED^FM and E3C, while CLaP performs better on MASSIVE.

[3] Contextual Label Projection for Cross-Lingual Structured Prediction

  • Core methodological distinction

We agree that quality- and difficulty-based data selection have a rich history; however, how these criteria should be designed and applied can differ across settings and downstream tasks, and this remains understudied for multilingual extraction. In this work, we investigate how these criteria should be applied in this setting.

Firstly, while components such as XCOMET are widely used to assess translation quality, it remains unclear whether they are sufficient for assessing the quality of translated training data for extraction tasks. In Section 5, we systematically study this question, and explain when XCOMET is sufficient and when it benefits from being complemented by an LLM judge.

Secondly, how to use the difficulty signal to select the data also needs to be studied in this setting. As shown in the previous point, directly applying D3, a selection method designed for open-ended instruction tuning, does not transfer well to extraction tasks and leads to oversampling of examples without labels. The reviewer’s suggested comparison with D3 has helped strengthen our analysis, providing further evidence that targeting a specific difficulty range is more suitable for extraction tasks. We have added this analysis to Appendix B.5.

Reviewer_Ji4cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates data selection for multilingual extraction tasks using machine-translated English training data. The authors argue that translation quality alone is insufficient because correctly translated examples may contain incorrectly transferred span annotations or provide limited learning value. The proposed approach combines three signals: translation quality measured using XCOMET-XXL, label transfer accuracy assessed by an LLM judge, and example difficulty estimated through the loss of a model fine-tuned on English data. Examples are selected using a quality-based score weighted by a Gaussian function that favors a target difficulty range. Experiments cover eight multilingual benchmarks and translations into 20 languages. The proposed strategy achieves better peak performance than quality-only selection on seven of eight benchmarks and shows improved data efficiency. The paper also demonstrates that fine-tuning on translated data can outperform few-shot prompting, even for relatively capable multilingual models.

Critical strengths

1.Important and well-motivated problem. The paper addresses a practical limitation of multilingual extraction: accurate sentence-level translation does not guarantee correct span-level supervision. The analysis in Section 5 provides useful evidence that label transfer accuracy can be more informative than standard translation metrics for certain languages and tasks.

2.Broad experimental evaluation. The study evaluates multiple extraction tasks, multilingual benchmarks, model sizes, and data budgets. The proposed approach achieves the best peak performance on 7 of 8 benchmarks and outperforms quality-only selection in 50 of 72 task-language comparisons at the 1x setting. Additional experiments with Llama-3.2 provide supporting evidence beyond the Gemma model family.

3.Practical improvements in data efficiency. The results suggest that carefully selecting translated examples can achieve competitive performance with fewer training instances. The comparison with few-shot prompting also demonstrates the continued value of multilingual fine-tuning, even when using models with strong existing multilingual capabilities.

Critical weaknesses

1.Limited methodological novelty. The central approach combines existing translation-quality metrics, LLM-based label verification, and loss-based difficulty estimation through a relatively simple Gaussian weighting function (Eq. 2). These underlying concepts have been explored in previous data selection research. Although their integration into multilingual extraction is useful, the paper does not establish a substantially new algorithmic principle. Comparisons with more advanced data selection approaches would better demonstrate the contribution beyond combining existing techniques.

2.Limited reliability of the LLM-based quality assessment. Label transfer accuracy is a central component of the proposed method, yet Appendix D reports only moderate agreement with human annotators (Cohen's kappa of approximately 0.54-0.56). Furthermore, this validation covers only Chinese, Vietnamese, and Korean. It remains unclear how reliably the judge evaluates span boundaries in other languages, particularly low-resource languages. Since selection explicitly prioritizes examples with perfect judged label transfer, these errors could directly influence the selected training data.

3.Insufficient experimental robustness and uncertainty analysis. Appendix B.4 indicates that all selection strategies except Random are evaluated using a single training run per configuration. Although the paper includes bootstrap confidence intervals and a statistically significant directional advantage across task-language pairs, these analyses do not capture variation across independent fine-tuning runs. In Appendix B.5, only 22 of 72 individual comparisons have strictly positive 95% confidence intervals. Additional training seeds would help establish whether the improvements are stable, particularly where performance differences are small. A related concern is that the claimed data efficiency does not account for the computational cost of translation-quality estimation, LLM-based label verification, and difficulty scoring.

Questions for authors

1.Novelty and stronger baselines: Can the authors compare their approach against more established difficulty-aware or information-based data selection methods under matched training-data budgets? This would help clarify whether the improvement comes from the proposed combination or from general principles already established in data selection research.

2.Reliability of the LLM judge: Given the moderate human agreement reported in Appendix D, how sensitive are the final results to errors in label transfer assessment? An ablation using human-verified labels or different judge models on a representative subset would strengthen confidence in this component.

3.Statistical robustness: Can the authors provide results across multiple independent fine-tuning seeds for the proposed method and quality-only baseline? This is especially important for datasets where the improvements are relatively small.

4.Computational efficiency and language coverage: How does the total cost of data scoring and selection compare with the training savings? Additionally, Table 3 includes evaluation languages that are not among the 20 translation targets listed in Section 4. How are these evaluation languages handled in the task-language selection and training setup?

4 comments

Thank you for your insightful feedback. Please find below our response to each point of your comments and some updates in the paper based on your suggestions.

Update on experimental results

While implementing the stronger baselines suggested by the reviewer, we identified an implementation issue in our difficulty-scoring pipeline: data shuffling in the dataloader caused the difficulty scores to be misaligned with the example IDs. This issue only affects the selection methods that use difficulty scores (e.g., Quality-Difficulty), while Random, Quality, and the correlation study in Section 5 are unaffected. We have fixed the issue and re-run all affected experiments with the same hyperparameters, and the main experiments in Figure 2 are now run with three random seeds. We have updated the paper accordingly. With the corrected scores, our main conclusion holds: Quality-Difficulty outperforms Quality on all 8 datasets, and is significantly better in 51 of the 72 task–language pairs. One change is that Difficulty-only now performs substantially below Quality. As we investigate in Appendix B.5, this is partly because prioritizing the highest-loss examples leads to over-selecting examples without labels, specifically in tasks where these examples are rare. This is consistent with our motivation for prioritizing examples within a target difficulty range rather than the most difficult ones.

  • Q1: stronger baselines

We have implemented D3 [1], a data selection method that jointly considers diversity, difficulty, and dependability. For a fair comparison, D3 uses the same English-only model as the seed model and Gemma-3-27B-IT as the teacher model, and follows the same per task–language selection setup. We compare the following selection methods: Quality, Quality-Difficulty, D3, and Difficulty-only (a simple baseline that prioritizes the examples with the highest loss under the English-only model). The table below reports the average performance (in %) over 8 test datasets of these selection methods at the 1$\times$ translation ratio. All experiments are run with 3 random seeds.

SelectionAverage score
Difficulty-only27.6 ± 0.5
D326.2 ± 1.0
Quality-Difficulty40.9 ± 0.5

From the table, we see that Difficulty-only and D3 achieve substantially lower average performance than Quality-Difficulty. One reason is that both methods prioritize the examples with high difficulty, which leads to over-selecting training examples with no label.For example, in BC5CDR, an English dataset included in our training data, only 16% of the examples have no labels. However, among the examples selected by D3 from the translated version of this dataset, 76% contain no labels. This substantial distribution shift leads the model to overpredict empty outputs: D3's model produces no labels for 25% of test instances that contain labels, compared with only 12% for the Quality-Difficulty method. This phenomenon can be attributed to two factors. First, in extraction tasks, the expected output of an instance with no label is an empty list, which means these instances always have perfect label transfer accuracy. Second, difficulty is defined as the mean loss per token. When the seed model incorrectly predicts label spans for an instance with no labels, the resulting loss per token tends to be high because the output contains only a few tokens. In contrast, errors on other instances are averaged over a larger number of tokens, resulting in lower mean loss per token. Unlike D3, which ranks examples higher as their difficulty increases, our method down-weights extremely difficult examples, which may be outliers. We have added this comparison to Appendix B.5.

[1] Jia Zhang et al., D3: diversity, difficulty, and dependability-aware data selection for sample-efficient llm instruction tuning. IJCAI 2025

  • Q2: Reliability of the LLM judge

First, we would like to clarify that the moderate kappa partially comes from class imbalance. The reported kappa is computed at the span level, where only 11.9% of spans are incorrect. With such imbalanced labels, kappa is low even though the LLM and humans agree on 91.4% (vi), 88.4% (zh), and 87.2% (ko) of the annotated label spans.

Secondly, the LLM judge is reliable for the decision used in data selection. Our selection method prioritizes perfect instances whose label spans are all correct. At this instance level, the LLM judge is accurate both when it accepts and when it rejects an instance:

Metricvizhko
Instances judged perfect by the LLM that are also perfect according to humans93.0%91.5%87.0%
Instances with incorrect labels (according to humans) that the LLM detects82.9%80.0%78.7%

Furthermore, regarding other languages, although we do not have human annotations for low-resource languages, our correlation study in Section 5 shows that the label transfer accuracy measured by the judge significantly correlates with downstream performance in several low-resource languages, such as Armenian, Khmer, Kannada, and Welsh (Table 1). This suggests that the judge also provides a meaningful signal in these languages, although we agree that a human validation in more languages would further strengthen this conclusion.

  • Q3: Statistical robustness

We re-ran the Quality and Quality-Difficulty experiments with three random seeds. The table below reports the average score (in %, mean ±\pm std over three seeds) of the two methods on each test dataset at the 1× translation ratio, together with the number of languages where Quality-Difficulty is significantly better (paired bootstrap test with 95% confidence intervals). We have updated Figure 2 in the paper to report the results averaged over three seeds at all translation ratios:

MethodIndicQARED^FMMASSIVEMultiCoNERE3CNaamapadamMu-SHROOMHate SpeechAVG
Quality34.2 ± 1.516.6 ± 0.235.0 ± 0.938.4 ± 2.349.3 ± 1.156.3 ± 0.448.8 ± 3.528.1 ± 1.238.3 ± 0.9
Quality-Difficulty38.4 ± 0.717.3 ± 0.337.9 ± 0.340.3 ± 0.751.7 ± 0.758.4 ± 0.752.0 ± 2.231.0 ± 2.040.9 ± 0.5
Significantly better11/112/612/157/113/46/98/132/351/72

Significantly better row shows for each dataset, the number of languages where Quality-Difficulty is significantly better than Quality.

With three seeds, Quality-Difficulty outperforms Quality on all 8 datasets, and is significantly better in 51 of the 72 task–language pairs

  • Q4 (Part 1): Computational efficiency

The table below shows the average performance (in %) over 8 test datasets of each selection method (Quality and Quality-Difficulty), and the GPU-hours (GPU-h) required for training and for preparing the selection, at different amounts of translated data. The performance is averaged over three random seeds. The one-time data preparation includes translation, XCOMET, and LLM-judge scoring, which are shared by both methods, while Quality-Difficulty additionally requires fine-tuning the English model and computing the loss of the translated data.

Translated dataTraining cost (GPU-h)QualityQuality-DifficultyΔ
0.25× (2.5k)4.936.9 ± 0.239.2 ± 0.3+2.3
0.5× (5k)5.937.7 ± 0.440.3 ± 0.5+2.6
1× (10k)7.838.3 ± 0.940.9 ± 0.5+2.6
2× (20k)11.738.8 ± 0.641.0 ± 0.6+2.2
One-time data preparation (GPU-h)—192.3218.4 (+26.1)—

Firstly, at the same data budget, Quality-Difficulty adds about 26.1 GPU hours, which is around 13.6% more in the one-time data preparation, and provides a gain of 2.2 to 2.6 points across data budgets. Secondly, for the same-performance comparison, Quality requires training with 20k translated instances, but its average performance is still lower than that of Quality-Difficulty selecting only 2.5k instances, which also reduces the training cost from 11.7 to 4.9 GPU hours per run. Therefore, although Quality-Difficulty requires additional data preparation, it brings consistent performance gains and requires less translated data for training. We have added this discussion in Appendix B.5.

  • Q4 (part 2): language coverage

In our evaluation datasets, there are 21 languages that are not among the languages we translate into and select from during training. No translated data is created or selected for these languages. They correspond to 23 of the 72 task-language pairs in the test sets and are therefore evaluated in a zero-shot cross-lingual setting. Among these 23 pairs, Quality-Difficulty is significantly better than Quality-only in 19 pairs and worse in none, suggesting that the benefit of the selection can transfer to languages unseen in the translated training data.

Thank you for the detailed rebuttal and the substantial effort invested in addressing my concerns. I particularly appreciate the transparency regarding the implementation issue and the additional experiments, which significantly strengthen the empirical contribution. Most of my original experimental concerns have been addressed, and I recognize the practical value of the proposed approach. However, my main concern regarding methodological novelty remains, as the method primarily combines existing data selection principles rather than introducing a substantially new mechanism. I currently find myself somewhere between a score of 2 and 3. I would be comfortable upgrading my score to 3, given the improved experimental evidence and the authors' constructive response. Nevertheless, I still consider the limited novelty a meaningful weakness, especially given the high bar for methodological contributions at ICLR.

Reviewer_aCi3OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper improves multilingual extraction by selecting translated data based on translation quality, label accuracy, and example difficulty. It beats Quality-only selection on 7 of 8 benchmarks, but gains vary across tasks and depend on an LLM judge.

Critical strengths
  1. Better data quality. Checking label accuracy helps catch errors that translation quality alone misses.
  2. Promising results. Quality-Difficulty wins in 50 of 72 task-language pairs and shows similar trends with Llama-3.2-3B-Instruct.
Critical weaknesses
  1. Limited scope. Experiments focus on extraction tasks, and the comparison with Gemma-3-27B uses few-shot prompting rather than fine-tuning.
  2. Unclear gains. Only 22 of 72 pairs show statistically significant improvements, and the LLM judge agrees with human annotators only moderately.
Questions for authors
  1. Larger model comparison: Can you compare fine-tuned Gemma-3-12B and 27B using the same data budget to separate the effects of model size and data selection?
  2. Practical gains: How large are the actual improvements over Quality-only, and are they worth the extra scoring costs?
  3. Task consistency: Which tasks and languages benefit most, given that only 22 of 72 pairs show statistically significant gains?
  4. Judge reliability: How much do errors in the LLM judge's label assessments affect the final results?
  5. Overall cost: After including translation, scoring, and fine-tuning, does Quality-Difficulty still offer better data efficiency?
2 comments

Thank you for your insightful feedback. Please find below our response to each point of your comments and some updates in the paper based on your suggestions.

Update on experimental results

After the submission, we identified an implementation issue in our difficulty-scoring pipeline: data shuffling in the dataloader caused the difficulty scores to be misaligned with the example IDs. This issue only affects the selection methods that use difficulty scores (e.g., Quality-Difficulty), while Random, Quality, and the correlation study in Section 5 are unaffected. We have fixed the issue and re-run all affected experiments with the same hyperparameters, and the main experiments in Figure 2 are now run with three random seeds. We have updated the paper accordingly. With the corrected scores, our main conclusion holds: Quality-Difficulty outperforms Quality on all 8 datasets, and is significantly better in 51 of the 72 task–language pairs. One change is that Difficulty-only now performs substantially below Quality. As we investigate in Appendix B.5, this is partly because prioritizing the highest-loss examples leads to over-selecting examples without labels, specifically in tasks where these examples are rare. This is consistent with our motivation for prioritizing examples within a target difficulty range rather than the most difficult ones.

  • Q1: Larger model comparison

We additionally fine-tuned Gemma-3-27B-IT at the 1$\times$ ratio of the translated data (10k instances) selected by the Quality-Difficulty method. The table below shows the average score (in %) over 8 test datasets, comparing few-shot prompting and the fine-tuned versions of three model sizes of the Gemma-3 family: 4B, 12B, and 27B. Due to computational constraints, we fine-tune the models and report the results with a single random seed:

ModelFew-shotFine-tunedΔ
Gemma-3-4B24.841.4+16.6
Gemma-3-12B34.446.6+12.2
Gemma-3-27B38.347.5+9.2

From the table, we can see that given the same data size, fine-tuning on the selected translated data still improves over the few-shot prompting baseline across all model sizes, including the 27B model (+9.2 points).

  • Q2 & Q5: Practical gains and Overall cost

The table below shows the average performance (in %) over 8 test datasets of each selection method (Quality and Quality-Difficulty), and the GPU-hours (GPU-h) required for training and for preparing the selection, at different amounts of translated data. The performance is averaged over three random seeds. The one-time data preparation includes translation, XCOMET, and LLM-judge scoring, which are shared by both methods, while Quality-Difficulty additionally requires fine-tuning the English model and computing the loss of the translated data.

Translated dataTraining cost (GPU-h)QualityQuality-DifficultyΔ
0.25× (2.5k)4.936.9 ± 0.239.2 ± 0.3+2.3
0.5× (5k)5.937.7 ± 0.440.3 ± 0.5+2.6
1× (10k)7.838.3 ± 0.940.9 ± 0.5+2.6
2× (20k)11.738.8 ± 0.641.0 ± 0.6+2.2
One-time data preparation (GPU-h)—192.3218.4 (+26.1)—

Firstly, at the same data budget, Quality-Difficulty adds about 26.1 GPU hours, which is around 13.6% more in the one-time data preparation, and provides a gain of 2.2 to 2.6 points across data budgets. Secondly, for the same-performance comparison, Quality requires training with 20k translated instances, but its average performance is still lower than that of Quality-Difficulty selecting only 2.5k instances, which also reduces the training cost from 11.7 to 4.9 GPU hours per run. Therefore, although Quality-Difficulty requires additional data preparation, it brings consistent performance gains and requires less translated data for training. We have added this discussion in Appendix B.5.

  • Q3: Task consistency

We re-ran Quality and Quality-Difficulty with three random seeds and repeated the paired bootstrap test at the 1× configuration. With three seeds, Quality-Difficulty is significantly better than Quality in 51 of the 72 task–language pairs, and significantly worse in only 1 pair. Regarding which languages benefit most, the improvements are most consistent on lower-resource languages: among the 29 pairs involving lower-resource languages, Quality-Difficulty is significantly better in 24. Most of the inconclusive pairs (14 of 20) occur on datasets with small test sets (Mu-SHROOM and REDFM) or on high-resource languages. The table below shows, for each dataset, the number of languages where Quality-Difficulty is significantly better than Quality:

IndicQARED^FMMASSIVEMultiCoNER v2E3CNaamapadamMu-SHROOMHate SpeechTotal
11/112/612/157/113/46/98/132/351/72

We have updated the analysis in Appendix B.5 accordingly.

  • Q4: Judge reliability

First, we would like to clarify that the moderate kappa partially comes from class imbalance. The reported kappa is computed at the span level, where only 11.9% of spans are incorrect. With such imbalanced labels, kappa is low even though the LLM and humans agree on 91.4% (vi), 88.4% (zh), and 87.2% (ko) of the annotated label spans.

Secondly, the LLM judge is reliable for the decision used in data selection. Our selection method prioritizes perfect instances whose label spans are all correct. At this instance level, the LLM judge is accurate both when it accepts and when it rejects an instance:

Metricvizhko
Instances judged perfect by the LLM that are also perfect according to humans93.0%91.5%87.0%
Instances with incorrect labels (according to humans) that the LLM detects82.9%80.0%78.7%

Furthermore, despite these errors, the label transfer accuracy measured by the LLM judge correlates moderately to strongly with downstream performance in our correlation study (Table 1), indicating that the judge provides a useful signal for the final results. Lastly, all selection strategies share the same LLM-judge scores, so they are compared on the same candidate pool with the same judge errors. We have added this analysis to Appendix D.

  • Extension beyond extraction tasks

Beyond extraction tasks, we also apply Quality-Difficulty to multilingual math reasoning: we translate the English GSM8K [1] training data (about 6.7k instances, including reasoning traces) into 10 languages. We measure the quality with XCOMET-XXL. Difficulty is computed as one minus the accuracy over 8 samples of Gemma-3-4B-IT fine-tuned on the English data only. We fine-tune Gemma-3-4B-IT, and evaluate on MGSM-Rev2 [2].The table below reports the average accuracy (in %) over the 10 languages. We vary the amount of translated data relative to the English data (1$\times$ means using 6.7k translated instances, the same as the amount of English data). At the 4$\times$ ratio, we run three random seeds and report the mean and standard deviation.

Selection0.5$\times$1$\times$2$\times$4 ×\times
Random66.468.369.470.5 ± 0.7
Quality64.465.367.469.8 ± 0.6
Quality-Difficulty66.668.269.271.7 ± 0.9

In this setting, translation quality alone is a weak indicator of training usefulness, as Quality falls below Random. Incorporating difficulty helps substantially: Quality-Difficulty outperforms Quality at all data budgets (by 1.8 to 2.9 points) and achieves the best result at 4$\times$. We have added this experiment to the Appendix B.5.

[1] Karl Cobbe et al., Training verifiers to solve math word problems, 2021.

[2] Jan-Thorsten Pete et al., Mind the gap... or not? how translation errors and evaluation details skew multilingual results, 2025.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Revisions

We thank all reviewers for their thoughtful reviews and constructive feedback. We are encouraged that the reviewers highlighted the following strengths of the paper:

  • Important and well-motivated problem: The paper addresses a practical limitation of multilingual extraction, where accurate sentence-level translation does not guarantee correct span-level supervision (Ji4c).
  • Better data quality: Checking label accuracy helps catch errors that translation quality alone misses (aCi3, Ji4c).
  • Effective across a broad evaluation: The study covers multiple extraction tasks, languages, model sizes, data budgets, and model families, and the proposed method, Quality-Difficulty, improves over Quality selection (aCi3, Ji4c, wY9B).
  • Simple and practical: The approach is easy to implement, improves data efficiency, and shows the continued value of fine-tuning on translated data (Ji4c, wY9B).

Following the reviewers' suggestions, we have made the following updates and additional analyses:

  • Statistical robustness (aCi3, Ji4c): The main experiments (Figure 2) are now run with three random seeds. Quality-Difficulty outperforms Quality in 68 of the 72 task–language pairs and is significantly better in 51. Among the 29 pairs involving lower-resource languages, Quality-Difficulty is significantly better in 24.
  • Comparison with established selection methods (Ji4c, wY9B): We compare Quality-Difficulty with D3, an established data selection method, and analyze why the proposed method outperforms D3 and is more suitable in the multilingual extraction setting.
  • Reliability of the LLM judge (aCi3, Ji4c): We clarify that the reported kappa is moderate because of class imbalance, and that the judge is highly accurate on the instance-level decisions used for selection (87–93% precision).
  • Computational cost (aCi3, Ji4c): Quality-Difficulty adds about 14% GPU-hours to the one-time data preparation; however, it is more data-efficient: fine-tuning on only 2.5k translated examples (0.25×) selected by Quality-Difficulty outperforms, on average, fine-tuning on 20k translated examples (2×) selected by Quality.
  • Larger models (aCi3): We additionally fine-tune Gemma-3-27B-IT on 10k (1×) translated examples selected by Quality-Difficulty, and the results show that training on translated data remains beneficial for larger models, improving over few-shot prompting by 9.2 points on average across the 8 test datasets.
  • Beyond extraction tasks (aCi3): We apply the proposed method to multilingual math reasoning (MGSM-Rev2) and find that Quality-Difficulty improves over Quality at all data budgets.
  • Translation strategies (wY9B): We have added a comparison between two label projection methods, CLaP and Mark-then-translate. Neither method is consistently better across tasks.
  • Zero-shot languages (Ji4c): We have added an analysis of evaluation languages that have no translated training data. Among the 23 task–language pairs involving these languages, Quality-Difficulty is significantly better in 19 and worse in none.

Correction. We also identified and fixed an implementation issue in the difficulty scoring pipeline; all affected experiments have been re-run, our main conclusions hold, and details are provided in each individual response.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.