acceptodds
Under review as a conference paper at ICLR 2027

FineWeb2-HQ-QA: Native Document-Grounded QA for Multilingual Pre-Training

Abstract

Recent advances in data curation have substantially improved the quality of multilingual web corpora, yet large-scale synthetic resources for non-English pre-training remain scarce and typically rely on machine translation from English-source data. We introduce *FineWeb2-HQ-QA*, a multilingual dataset containing 193B tokens of source documents and generated QA across 20 languages. We generate QA pairs directly in each target language from native FineWeb2-HQ web documents rather than from English-source text (*native QA*). This approach is designed to reduce dependence on English-source topic distributions by grounding generation in target-language web content. At 3B and 8B scales, we compare our native document-grounded QA against non-QA baselines and Nemotron's English and machine-translated QA. Relative to matched non-QA baselines, our dataset improves Global and Local Knowledge across all four training settings while largely preserving or improving aggregate general language understanding. Among individual QA sources, translated QA yields the largest observed Global Knowledge gains across all four settings, whereas native QA consistently shows higher observed scores than translated QA on general language understanding and achieves strong Local Knowledge gains at 8B. In natural-mixture settings, combining sources yields a more balanced profile across categories without consistently exceeding the best individual source per category. We release our dataset alongside our generation and filtering pipelines to support future work on multilingual synthetic pre-training data.

Primary area:
datasets and benchmarks
Keywords:
multilingual pre-training data, synthetic data generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YMs1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a multilingual pre-training resource built by generating questions and answers directly from target-language web documents, called FineWeb2-HQ-QA. The authors evaluate whether this document-grounded QA provides useful training signal compared with the original documents, English-source QA, and translated QA. The results suggest that native QA is a useful complement to existing resources, with different trade-offs across knowledge and general language understanding rather than consistent superiority over translated data.

Critical strengths
  1. The paper contributes a useful alternative to English-centered synthetic data. Importantly, the design choice to generate QA pairs from target-language documents allows the resource to draw on content that may be less visible in English-source corpora. The value of the dataset is also shown by downstream training experiments.
  2. The paper is useful as a study of complementary multilingual data sources. The authors show that the translated QA pairs remains stronger in some settings and that combining resources does not always improve performance.
Critical weaknesses
  1. The connection between target-language sourcing and locally relevant knowledge is confusing. While the motivation of the paper suggests that this approach can recover content overlooked by English-centered resources, the actual experiments rather show downstream usefulness than they establish what distinctive knowledge the dataset contributes. The paper lacks a closer analysis of the content coverage, which makes it difficult to tell whether the dataset reflect locally specific information.
  2. The qualitative assessment of the dataset also is lacking. Although the conducted automated filtering provide useful checks, the QA is still generated by a single model. This is concerning since it may introduce generation that favor familiar facts, standardized language, and conventional question types.
Questions for authors
  • How did the authors ensure or measure what is lost or changed when native documents are converted into synthetic QA? For example, have the authors looked into whether generation preserves culturally specific content and linguistic variation?
  • Will users be able to trace generated QA back to its source and inspect why a sample was retained or removed?
  • How did the authors distinguish useful QA augmentation from repeatedly restating information already present in the source?
  • How can we ensure that the automated quality judgments reflect readers’ judgments in the target languages?
3 comments

We thank the reviewer for the constructive questions on local content coverage, traceability, QA augmentation vs. restatement, and human validation of the automated audit. Please see the General Comment for the experimental setup and results of our 13-language Wikidata entity analysis (Section 2) and two-stage human evaluation (Section 3). Below we address your four questions directly.

Q1. Measuring What Is Preserved or Lost When Converting Native Documents to QA (W1 & Q1)

We measured local content coverage and preservation at two levels (General Comment Sections 2-3):

  1. Corpus-wide locality across 13 languages: Across 260,000260{,}000 samples in each dataset, FineWeb2-HQ-QA has a much higher share of target-country over US/UK entity mentions than TDQA in all 13 BM languages (e.g., 49.7% vs. 6.1% in German, 41.8% vs. 7.2% in French, 68.0% vs. 7.6% in Japanese).

  2. Fact preservation and naturalness in QA generation: In Stage 1 of our human evaluation (blind to QAs), the annotator pre-registered 48 local facts across the locally anchored German (22%, DACH) and French (10%, Francophonie) documents. Stage 2 shows that 87.5% (42/48) of these facts are fully (22/48) or partially (20/48) preserved in the generated QAs. Overall, 83.2% of all 798 QAs were rated natural. While 30%-40% of source web documents in German and French showed translationese or non-fluent phrasing, this compares to 98% (49/50) with translationese artifacts in German TDQA.

Q2. Traceability to Source Documents and Filter Inspection (Q2)

  • Yes: Every sample in FineWeb2-HQ-QA pairs the original FineWeb2-HQ source document directly with its generated QA pairs.
  • As stated in Section 3.3 (Release), we release the unfiltered 188.8M-document dataset together with our deterministic datatrove 4-stage filtering pipeline (completed </think> check, GlotLID, Gopher repetition, and 13-gram decontamination). Users can trace every QA pair back to its source passage, deterministically reproduce the 174.0M-document filtered dataset (and inspect which stage removed any sample), or customize per-language filter thresholds.

Q3. Distinguishing Useful QA Augmentation from Restating Source Information (Q3)

  • Conceptually, Allen-Zhu & Li (2024) show that pre-training on raw passages alone often fails to make facts extractable at test time unless the knowledge is rewritten in diverse forms, and Su et al. (2025) and Niklaus et al. (2026) find document-grounded QA rephrasing more effective than prose rephrasing.
  • QA augmentation vs. synthetic rephrasing (supporting ablation, Appendix C, Table 3): In a separate 3B cooldown experiment on Nemotron-CC-v2 subsets, the rephrased High-Quality-Synthetic subset changes the average by only +0.4+0.4 / −0.5-0.5 pp (English / multilingual) over the baseline, whereas Diverse QA improves it by +6.1+6.1 / +1.7+1.7 pp. This motivated our choice of QA augmentation over prose rephrasing. It does not involve FineWeb2-HQ-QA itself.
  • Reformulation rather than restatement: Our quality audits indicate that the QAs reformulate rather than merely restate the source: only 0.5% of QAs in our human audit are seriously trivial/tautological (3.0% including minor shallow questions).

Q4. Target-Language Quality Judgments & Single-Model Generation (W2 & Q4)

  • Human validation of the automated audit: On 798 German and French QAs (159 double-annotated; General Comment (Section 3, Table 2), Gemini 3.6 Flash closely matches humans on serious defects (7.1% vs. 7.6%, κ=0.78\kappa = 0.78; two-annotator κ=0.87\kappa = 0.87). While standalone κ\kappa is lower on the rare Serious other (κ=0.37\kappa = 0.37, 96.7% agreement), 14 of the 24 human-only cases are already caught by Gemini as serious unfaithful or unanswerable rather than double-tagged. Across all 6 audited languages, repeat-run agreement is high (κ=0.76-0.84\kappa = 0.76\text{-}0.84) and defect-injection recall is 99.5% (1,075/1,0801{,}075 / 1{,}080). Because Gemini under-flags minor issues and its document-level factual-error flags do not match humans (κ=0.13\kappa = 0.13), we will present the judge as validated for serious QA-level defects only (Section 3.3, Appendix F).
  • Single-model generation (Qwen3-30B-A3B): We used Qwen3-30B-A3B, the same model used to translate Nemotron DQA into TDQA (Section 3.1). While it preserves 87.5% (42/48) of pre-registered local facts (fully or partially) and produces 83.2% natural QAs across six prompted question formats, prompting a single model for "clear and concise" QAs favors standard written register over dialectal variation and can standardize phrasing. We will note this in Limitations (Section 6 / Appendix A).
Reviewer_YMs1OpenReview ↗

Response to the Authors

Thank you for the response and for clarifying the scope of the comparison! I see FineWeb2-HQ-QA as an useful multilingual resource (especially playing a role as an alternative to more of English-centered synthetic data), and the experiments show that it can improve knowledge scores over the matched document-only baseline. My main concern remains that the comparison with traslated QA cannot isolate the effect of generating QA directly in the target language, since the source corpora and processing pipeline also differ. (Although the authors mention that running a fully controlled comparison is costly, I feel this is one of the most critical piece in the work to claim the contribution that it's making.) The revised framing makes that distinction clearer, but it does not change my overall assessment. I'll retain my original score.

Thank you for the constructive discussion and for recognizing FineWeb2-HQ-QA as a useful multilingual resource. We agree that our comparison with Nemotron TDQA does not isolate the effect of target-language generation, as noted explicitly in Section 1. As a dataset paper, our goal is to release an open 193B-token, 20-language pre-training resource, show consistent gains over the matched document-only baseline and mixing trade-offs across four 3B/8B settings, and (added in the revised PDF) characterize its target-region coverage, local-fact preservation, and translationese compared with TDQA (Appendix G), as well as its human-validated quality (Appendix F).

Reviewer_B5E3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FineWeb2-HQ-QA, a multilingual pre-training dataset containing approximately 193B tokens across 20 languages. Rather than translating English-generated question-answer pairs, the authors generate QA pairs directly from target-language FineWeb2-HQ documents using Qwen3-30B-A3B, followed by a four-stage filtering pipeline. The approach is evaluated through continued pre-training experiments with 3B and 8B parameter models, comparing native QA against document-only baselines and English or translated QA from Nemotron. Results show consistent improvements in Global and Local Knowledge over the non-QA baselines. Translated QA generally performs better on Global Knowledge, while native QA achieves competitive general language understanding and strong Local Knowledge improvements, particularly at 8B.

Critical strengths

1. Large-scale multilingual dataset contribution (Section 3). The paper introduces a substantial resource containing approximately 174 million retained document-grounded QA samples across 20 languages. Generating QA directly from target-language documents addresses an important limitation of English-centric synthetic data. The dataset and accompanying generation pipeline could provide meaningful practical value to the multilingual pre-training community. 2. Comprehensive resource-level evaluation (Sections 4 and 5). The evaluation includes two model scales (3B and 8B), balanced and natural language mixtures, and comparisons against document-only, English QA, and translated QA baselines. The experiments demonstrate consistent knowledge improvements over non-QA baselines and reveal useful trade-offs between global knowledge, local knowledge, and general language understanding. The inclusion of a detailed automated quality audit further strengthens the empirical contribution.

Critical weaknesses

1. Limited methodological novelty and insufficiently controlled comparisons (Sections 2, 3.1, and 5). The proposed approach largely reuses existing techniques, including Nemotron's QA generation strategy, an existing multilingual corpus, and standard filtering methods. Similar target-language generation approaches have already been explored, including in Nemotron Nano 2. Consequently, the main novelty lies in dataset scale rather than a fundamentally new methodology. More importantly, the comparison between native and translated QA uses different source corpora and generation pipelines. As acknowledged by the authors, this prevents isolating whether the observed differences result from native-language generation, source quality, topical coverage, or knowledge density. A controlled comparison using matched source content and generation settings would substantially strengthen the central contribution.

2. Limited statistical reliability and incomplete validation of synthetic data quality (Sections 3.3, 4.2, and Appendix F). Each training configuration is evaluated using only a single training run, and the reported error bars capture evaluation uncertainty rather than training variability. This limits confidence in smaller performance differences, particularly for general language understanding. Furthermore, the dataset quality assessment relies on an automated LLM judge without independent human validation. Although the authors evaluate judge consistency and sensitivity to injected errors, the audit still identifies approximately 5.3-7.0% unfaithful answers across sampled languages. The filtering pipeline does not directly verify factual faithfulness, raising concerns about residual noise in the released training data.

Questions for authors

1. Controlled comparison: Can the authors provide an experiment comparing native QA generation and translated QA using matched source content, the same generation model, and comparable filtering? This would help determine whether native generation itself contributes to the reported improvements.

**2. Statistical reliability: Can the authors provide results across multiple training seeds for at least a subset of configurations? This would clarify whether the relatively small GLU differences are robust to training variability.

3. Dataset quality: Can the authors provide a small human-annotated evaluation of QA faithfulness and factual correctness, or demonstrate whether filtering unfaithful QA pairs affects downstream performance?

3 comments

We thank the reviewer for the constructive review. Please see the General Comment for our resource-level comparison rationale (Section 1), the 13-language Wikidata entity analysis (Section 2), and our two-stage human evaluation (Section 3). Below we answer your three questions.

Q1. Resource Novelty & Controlled Comparison of Native vs. Translated QA (W1 & Q1)

As noted in General Comment (Section 1), our contribution is an open 193.2B-token web-grounded pre-training resource and mixture study rather than a new QA prompt. Running 50B-63B token cooldowns on translated FineWeb2-HQ or English FineWeb-HQ would require generating and training on tens of billions of tokens of artificial intermediate data. Instead, we compare these two datasets directly at the data level (General Comment Sections 2-3):

  1. Across 260,000260{,}000 samples in all 13 BM languages in each dataset, FineWeb2-HQ-QA has a substantially higher share of target-region vs. US/UK entity mentions than TDQA in all 13 languages (e.g., 49.7% vs. 6.1% in German, 41.8% vs. 7.2% in French, 68.0% vs. 7.6% in Japanese).
  2. In our human evaluation (N=150N=150 documents), 98% (49/50) of German TDQA samples exhibit translation artifacts and 0/50 contain DACH local facts (instead importing translated Canadian/UK/US facts), whereas FineWeb2-HQ-QA QAs are 83.2% natural and fully or partially preserve 87.5% (42/48) of pre-registered regional facts.

Q2. Statistical Reliability Across Training Runs and Paired SEs (W2 & Q2)

  • Multi-seed 50B-63B token continual pre-training runs across 3B and 8B models exceed our compute budget. We agree that the paired standard errors below capture evaluation-set sampling only, not run-to-run training variance. As partial evidence, the direction of the main effects is consistent across our 2 scales (3B, 8B)×2 mixtures (BM, NM)=42 \text{ scales (3B, 8B)} \times 2 \text{ mixtures (BM, NM)} = 4 setups, which differ in starting checkpoint, model size, and language mixture:
    • Knowledge gains over baseline: Ours improves over the non-QA Baseline in 4/4 setups on both GlobalMMLU (+4.84±2.13+4.84 \pm 2.13, +5.60±1.51+5.60 \pm 1.51, +2.69±1.23+2.69 \pm 1.23, +2.53±0.99+2.53 \pm 0.99 pp) and INCLUDE (+2.34±0.81+2.34 \pm 0.81, +1.49±0.34+1.49 \pm 0.34, +3.68±0.64+3.68 \pm 0.64, +3.38±0.30+3.38 \pm 0.30 pp).
    • GLU consistency (Ours vs. TDQA): Ours scores higher than TDQA on GLU Avg in 4/4 setups, with the paired difference exceeding 3.23.2 SE in three of the four setups: +1.47±0.30+1.47 \pm 0.30 pp in 3B NM (4.84.8 SE), +0.93±0.27+0.93 \pm 0.27 pp in 8B BM (3.43.4 SE), and +0.57±0.17+0.57 \pm 0.17 pp in 8B NM (3.33.3 SE), alongside +0.39±0.45+0.39 \pm 0.45 pp in 3B BM.

Q3. Human-Annotated Evaluation of QA Faithfulness and Correctness (W2 & Q3)

Following your suggestion to provide a human-annotated evaluation of QA faithfulness, we conducted a two-stage human audit of 100 documents (798 QA pairs) in German and French (50 documents in each language), with a 20-document (159 QA) double-annotated German overlap (General Comment Section 3, Table 2):

  • Defect rates: Human annotators found 92.4% of QAs free of serious defects, with 4.9% serious unfaithful answers and 4.4% serious unanswerable questions, close to Gemini 3.6 Flash on the same QAs (5.1% and 3.3%).
  • Agreement: On serious defects, human-Gemini agreement (κ=0.78\kappa = 0.78, 97.0% agreement) is close to agreement between the two annotators (κ=0.87\kappa = 0.87). While standalone κ\kappa is lower on the rare Serious other pool (κ=0.37\kappa = 0.37, 96.7% agreement), 14 of the 24 human-only cases are already caught by Gemini as serious unfaithful or unanswerable (since its prompt instructs it not to double-tag overlapping defects, whereas humans often co-labeled both). Gemini flags far fewer minor issues than humans (e.g., 6.5% vs. 22.1% unfaithful including minor cases), so we will report the paper's automated audit rates as serious-defect rates (Section 3.3, Appendix F).
  • Residual ~5% serious unfaithful QAs: Running an LLM faithfulness verifier across all 188.8M documents (~1.5B QA pairs) is computationally expensive at pre-training scale, and Section 6 already names a verification step as future work. Despite ~5% serious unfaithful QAs, Ours improves knowledge over the document-only Baseline in all 4 setups, and releasing line-delimited QAs paired with each source passage allows future work to apply lightweight QA-level verification.

Thank you for the detailed response and additional analyses. I appreciate the effort to address my concerns, particularly the new evidence regarding cultural knowledge and filtering. These clarifications are helpful, although some limitations remain, especially the lack of a fully controlled comparison between native and translated QA, the absence of multiple independent training runs, and the limited human validation of generated data quality. I will carefully review your responses and the additional evidence before making my final assessment.

Thank you for engaging with our response. We have uploaded the revised PDF (changes in blue). A fully controlled training comparison would require regenerating and training on tens of billions of tokens of matched translated data, which was beyond our compute budget. Instead, we characterize how native and translated QA differ directly at the data level: native QA has a higher target-region entity share in all 13 languages (Appendix G.1), preserves 87.5% (42/48) of pre-registered local facts, and shows translationese or unnatural phrasing in 30–40% of native source passages (DE/FR) vs. 98% of German TDQA passages (Appendix G.2). While each configuration was trained once, knowledge gains over the document-only baseline hold in all four configurations (>2.1 standard errors), and native QA scores higher than TDQA on GLU Avg in all four (>3.2 standard errors in three; Appendix E.4). Finally, our human evaluation covers 798 QAs in German and French (including a 159-QA double-annotated subset) and validates the automated audit on serious defects (κ = 0.78 vs. κ = 0.87 between annotators; Appendix F.1), supporting our 6-language audit of ~12K QAs (Appendix F).

Reviewer_TzM8OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

FineWeb2-HQ-QA extends FineWeb2-HQ by adding question-answer (QA) pairs generated by Qwen3-30B-A3B to documents in their original languages. The resulting dataset contains around 174 million document-QA samples and 193.2 billion tokens, including both the documents and their associated QA content, across 20 languages. Continual pre-training Experiments with 3B and 8B models compare native-language QA against document-only data, English QA (DQA), translated QA (TDQA), and different data mixtures under balanced and natural language distributions. Native-language QA improves both knowledge categories across all four settings compared with the document-only baseline. TDQA achieves the highest observed Global Knowledge scores among the individual data sources, while native-language QA performs better than TDQA on benchmark-averaged General Language Understanding (GLU). Overall, no single data source consistently performs best across all evaluations.

Critical strengths
  • Potentially useful data resource contribution
    The native-curation of Q/A pairs can serve as a rich resource on data mixing / continual learning strategies .

  • Complementary over a sufficiently pre-trained checkpoint
    When continuing an 8B checkpoint already trained on 9.5T tokens: native QA improves Global Knowledge by 2.7/2.6 percentage points and Local Knowledge by 3.6/3.4 points in balanced/natural mixtures, respectively

Critical weaknesses
  • Insufficient evidence to prove that native sourcing is necessary
    The experiments show that native QA improves performance over document-only training, but do not establish a consistent advantage from sourcing QA in the original language. The results vary by task and model size: TDQA performs better than native QA on Global Knowledge in all four settings, while native QA achieves higher average GLU scores. The ranking between native QA and TDQA on INCLUDE also reverses between the 3B and 8B models, and English-only DQA outperforms native QA in three of the four settings (Tables 5–8). These findings demonstrate trade-offs between data sources rather than consistent superiority of native-language content. Although the authors acknowledge that the study does not establish causality, they do not sufficiently explain when and why native sourcing is beneficial. More controlled comparisons of source content and quality, direct analysis of additional local-knowledge coverage, and uncertainty estimates for comparisons between data sources are needed to establish the specific benefits of the proposed sourcing strategy.

  • Native-grounded QA generation is not, by itself, a new contribution.
    Section 3.1 adapts an existing QA prompt from Nemotron, and Section 2 acknowledges prior work on target-language Wikipedia QA, including Nemotron Nano 2. UPDESH also explicitly compares bottom-up instruction generation grounded in language-specific Wikipedia with top-down translation from English. Although UPDESH focuses on instruction tuning rather than pre-training, it provides a relevant conceptual precedent, so the authors should compare it with this work, positioning their contribution as a large-scale, multilingual, web-grounded pre-training resource rather than a new data-sourcing principle. More importantly, the paper should demonstrate what additional target-community knowledge this resource actually captures.

  • The translation comparison overlooks filtering and source content
    The translation comparison overlooks filtering and source content. Doshi et al. (2024) show that translating English documents and filtering them using small language models trained on clean target-language text can achieve competitive results in Indic-language pre-training. Though focused on documents rather than QA, this work is relevant to the paper's translationese motivation and should be discussed. Their use of English documents targeting Indic content also highlights that source language and regional content are distinct factors. Since the current study jointly varies source selection, generation, and filtering, and does not report QA-only token exposure, equal total tokens alone cannot establish a native-sourcing advantage. Content- and quality-matched comparisons are needed.

  • Evaluation data provenance is not sufficiently controlled.
    Several multilingual benchmarks, including GlobalMMLU, XNLI, PAWS-X, and m_ARC/m_HellaSwag, rely on English-origin content, and prior work shows that translation artifacts can affect evaluation results Uppadhyay et al., COLM 2026. This is particularly relevant to a paper motivated by data provenance, as benchmark rankings may reflect alignment with English-source topics or translation patterns rather than data quality alone. However, this does not establish that TDQA is favoured: native QA achieves higher average GLU scores in all four settings, and INCLUDE includes locally sourced content. Still, English-only DQA outperforms native QA on INCLUDE in three of four settings (Tables 5-8), leaving the native-sourcing advantage unclear. Evaluation on natively authored, culturally relevant content and target-language-generated tasks would test the paper's motivation more directly. The culturally sensitive/agnostic split in GlobalMMLU could provide an additional diagnostic.

  • Data quality
    The pipeline uses a common LID cutoff of 0.9 and repetition cutoffs of 0.25/0.1 (Appendices D.2-D.3). It removes 16.96% of initial Arabic samples through LID and 37.68% of initial Japanese samples through repetition filtering. These fixed thresholds may not be equally well calibrated across languages or generalise equally well to native and translated data. DCAD-2000 (Shen et al., 2025) directly motivates moving beyond fixed feature thresholds. The authors should include threshold-sensitivity or filtering ablations to clarify how these choices affect data quality and downstream performance, and provide practical guidance on filtering multilingual data.

Questions for authors

Please check weaknesses section.

2 comments

We thank the reviewer for the detailed feedback and references. Please see the General Comment for our discussion of resource-level comparisons (Section 1), the 13-language Wikidata entity locality analysis (Section 2), and the human evaluation comparing FineWeb2-HQ-QA and TDQA (Section 3). Below we address each remaining point.

1. When and Why Native Sourcing Helps & Paired Between-Source Uncertainty (W1)

  • Data-level evidence: As shown in General Comment (Sections 2-3), native sourcing yields a much higher share of target-region entity mentions across all 13 BM languages, fully or partially preserves 87.5% (42/48) of pre-registered local facts in the QAs, and avoids the 98% translationese rate found in German TDQA.
  • Paired between-source standard errors: Using the same item-clustered estimator as Figure 1 (clustering translations of the same source item across languages), the table below reports paired differences (±1\pm 1 standard error in percentage points) between sources:
SetupContrastΔ\Delta GlobalMMLU (±\pmSE)Δ\Delta INCLUDE (±\pmSE)Δ\Delta GLU Avg (±\pmSE)
3B BMOurs −- Baseline+4.84 ±\pm 2.13+2.34 ±\pm 0.81-0.25 ±\pm 0.40
Ours −- TDQA-7.97 ±\pm 2.39-5.61 ±\pm 0.83+0.39 ±\pm 0.45
Ours −- DQA-3.56 ±\pm 1.78-5.95 ±\pm 0.82+0.62 ±\pm 0.47
3B NMOurs −- Baseline+5.60 ±\pm 1.51+1.49 ±\pm 0.34+1.33 ±\pm 0.31
Ours −- TDQA-4.80 ±\pm 2.06-2.99 ±\pm 0.37+1.47 ±\pm 0.30
Ours −- DQA-2.42 ±\pm 1.62-4.27 ±\pm 0.39+2.13 ±\pm 0.29
8B BMOurs −- Baseline+2.69 ±\pm 1.23+3.68 ±\pm 0.64+0.89 ±\pm 0.24
Ours −- TDQA-1.88 ±\pm 1.18+1.39 ±\pm 0.62+0.93 ±\pm 0.27
Ours −- DQA+0.91 ±\pm 1.10+0.92 ±\pm 0.58+0.13 ±\pm 0.25
8B NMOurs −- Baseline+2.53 ±\pm 0.99+3.38 ±\pm 0.30+0.47 ±\pm 0.15
Ours −- TDQA-1.62 ±\pm 1.02+0.30 ±\pm 0.28+0.57 ±\pm 0.17
Ours −- DQA+0.53 ±\pm 1.01-0.21 ±\pm 0.28+0.17 ±\pm 0.17
  • Interpretation of trade-offs: FineWeb2-HQ-QA (Ours) improves over the document-only Baseline on GlobalMMLU (≥2.2\ge 2.2 SE) and INCLUDE (2.9-11.32.9\text{-}11.3 SE) in all four setups, and outperforms TDQA on GLU Avg by >3.2>3.2 SE in three of four setups (3B NM, 8B BM, 8B NM). At 3B, starting from a weak 200B-token checkpoint that still benefits from general multiple-choice formatting (Appendix C), English/translated QA (DQA/TDQA) drives larger exam gains, including on INCLUDE. At 8B (cooled down from 9.5T tokens), English DQA no longer clearly outperforms Ours on INCLUDE (+0.92±0.58+0.92 \pm 0.58 pp in BM; −0.21±0.28-0.21 \pm 0.28 pp in NM, within 11 SE, where NM's include_base_44 averages over 44 languages, most of which are not among Ours' 20 languages), the GlobalMMLU gap to TDQA narrows to within 1.61.6 SE, and Ours achieves the highest INCLUDE score in BM (the 11 BM languages covered by INCLUDE; +1.39±0.62+1.39 \pm 0.62 pp over TDQA, 2.22.2 SE). We will add these paired SEs to Appendix E.

2. Positioning vs. UPDESH & Target-Community Knowledge (W2)

  • Our manuscript already adopts this positioning and does not claim a novel generation method:
    • In Section 3.1 (QA Augmentation), we explicitly state that "we adopt the QA generation technique from Nemotron-CC (Su et al., 2025)" and adjust its prompt template to generate QA pairs in each target language, rather than proposing a new generation method.
    • In Section 2 (Multilingual Synthetic Augmentation), we already credit prior target-language synthetic work, including MURI-IT (Köksal et al., 2025), SeaLLMs (Nguyen et al., 2024; Ng et al., 2025), and Nemotron Nano 2's target-language Wikipedia QA (Basant et al., 2025), and state that "our work extends this direction with a 193B-token, 20-language dataset of source documents and QA pairs grounded in FineWeb2-HQ web documents."
    • In Section 1 (Introduction), we also explicitly note that "language-based sourcing does not guarantee geographic, cultural, or dialectal representativeness or that every source document was originally authored in that language."
  • Where we will make this clearer: We will add UPDESH in Section 2 as a Wikipedia-based instruction-tuning precedent, and make it more prominent in Sections 1 and 2 that while Nemotron Nano 2 explored target-language Wikipedia QA internally, nvidia/Nemotron-CC-v2 publicly released only the English (Diverse-QA) and translated (Translated-Diverse-QA) web subsets, making FineWeb2-HQ-QA, to our knowledge, the first publicly available, large-scale pre-training dataset of QA generated directly from target-language web documents. Target-community coverage in our dataset is now quantified in General Comment (Sections 2-3).

3. Doshi et al. (2024) & QA-Only Token Exposure (W3)

  • We will discuss Doshi et al. (2024) in Section 2 regarding filtered machine-translated pre-training data. We also agree that source language and regional content are distinct factors (as noted in our Section 1 caveat). In our human evaluation, 22% of German and 10% of French web documents are labelled locally anchored, while the rest cover general topics. At the same time, comparing the two resources directly shows that FineWeb2-HQ-QA still contains a higher share of target-region entity mentions than TDQA across all 13 BM languages (12.4%-68.0% vs. 0.4%-14.8%; General Comment Section 2), i.e., grounding generation in target-language web documents yields a different content distribution than translating English-source QA.
  • Share of QA tokens vs. document tokens: Because each training sample concatenates a source passage with its generated QAs, we measured the share of tokens coming from the QA pairs across ∼10,000\sim 10{,}000 samples per language in both FineWeb2-HQ-QA and TDQA:
    • BM (13 shared languages): QA pairs account for 28.5% of tokens in FineWeb2-HQ-QA vs. 24.0% in TDQA (the remaining 71.5% and 76.0% are source document tokens).
    • NM (all languages): QA pairs account for 27.3% of tokens in FineWeb2-HQ-QA vs. 24.3% in TDQA.
    • Both datasets thus have a very similar QA-to-document token ratio (slightly higher in FineWeb2-HQ-QA), ruling out higher QA token share as the cause of TDQA's GlobalMMLU scores.

4. Translated Benchmarks (Uppadhyay et al., 2026) & GlobalMMLU CS/CA Split (W4)

  • We will cite Uppadhyay et al. (COLM 2026) in Section 4.2 when discussing translation artifacts in English-origin benchmarks.
  • GlobalMMLU Culturally Sensitive (CS) vs. Culturally Agnostic (CA) split: Following your suggestion, we disaggregated GlobalMMLU by official CS and CA item labels (macro-averaged over languages, with item-clustered paired SEs):
SetupBaseline (CS / CA)DQA (CS / CA)TDQA (CS / CA)Ours (CS / CA)Ours −- TDQA Δ(CS−CA)\Delta(\text{CS} - \text{CA}) (±\pmSE)
3B BM25.2 / 26.633.4 / 35.237.7 / 39.831.4 / 30.1+3.31 ±\pm 4.79
3B NM25.2 / 24.332.2 / 33.334.7 / 35.629.5 / 31.3-0.93 ±\pm 4.13
8B BM52.8 / 56.753.9 / 59.256.7 / 61.954.5 / 60.4-0.75 ±\pm 2.36
8B NM48.3 / 51.849.5 / 54.551.7 / 56.749.9 / 55.2-0.30 ±\pm 2.04
  • Models tend to score higher on CA than CS items, and Δ(CS−CA)\Delta(\text{CS} - \text{CA}) between Ours and TDQA (and DQA) is within 11 SE of zero in all four regimes. One interpretation of why native sourcing shows no relative gain on GlobalMMLU's CS subset is that "Culturally Sensitive" in GlobalMMLU marks items tied to any culture rather than the target language's culture: it retains MMLU's US-centric subjects (e.g., US History, US Foreign Policy, US law), and 84.9% of its geographically tagged CS items focus on North America or Europe (Singh et al., 2025). Consequently, GlobalMMLU's CS subset largely tests Western/US cultural knowledge (which aligns with English-sourced TDQA), whereas target-region knowledge is tested by the natively authored exams in INCLUDE. We will add this diagnostic to Appendix E.

5. Filtering Thresholds, Score Distributions & DCAD-2000 (W5)

  • Score distributions & raw release: We run LanguageFilter (GlotLID ≥0.9\ge 0.9) and GopherRepetitionFilter (≤0.25\le 0.25 nn-gram / ≤0.10\le 0.10 line, language=iso_code) on the QA block alone before merging with the source passage. Appendix D.2-D.3 (Figures 2-3) already reports per-language score distributions and notes that fixed global cutoffs over-remove Arabic in LID (16.96%) and Japanese/Chinese in repetition (37.68% / 10.76%, vs. <1.9% elsewhere), which is why Section 3.3 releases the unfiltered 188.8M-document split alongside our filtering code. Post-filter serious-defect rates are consistent across all 6 audited languages (5.4%-8.3%).
  • Why fixed cutoffs miscalibrate & practical guidance: Beyond genuinely degenerate QAs (Appendix D.4), repeated English - Question: / Answer: labels make up 27%-31% of non-whitespace QA characters in compact CJK scripts (jpn_Jpan, cmn_Hani) vs. 14%-20% in alphabetic scripts, which likely inflates character-weighted nn-gram repetition scores in CJK; for Arabic, short QAs with names, numerals, and these English labels may lower GlotLID confidence. In the revision, we will cite DCAD-2000 (Shen et al., 2025) as a principled multi-feature alternative to fixed global cutoffs and suggest stripping English Question: / Answer: labels before running LID and nn-gram repetition filters (which users can directly apply using our released raw split and deterministic datatrove pipeline).

Comments on the paper

4 comments

We thank the reviewers for their feedback and for recognizing FineWeb2-HQ-QA as a useful multilingual pre-training resource. Across the reviews, three common questions were raised:

  1. Resource-level comparison vs. controlled ablations (R1, R2): Why do our pre-training experiments compare complete public resources rather than varying single pipeline steps?
  2. Local knowledge coverage and provenance (R1, R2, R3): What target-language local knowledge and linguistic properties does sourcing from native web documents capture compared to translated English QA (TDQA), and what is preserved during QA generation?
  3. Human validation of QA quality and the LLM judge (R2, R3): How well do the generated QAs and the automated Gemini 3.6 Flash audit align with human judgments in target languages?

Below we address these three questions in order: Section 1 explains our resource-level evaluation scope and public dataset gap, Section 2 presents a 13-language Wikidata entity locality analysis (260,000260{,}000 samples each), and Section 3 presents a two-stage human evaluation (50 German and 50 French documents with 798798 QAs, plus 5050 German TDQA documents) covering both local-fact/translationese differences and LLM-judge validation. We also report paired between-source standard errors, QA-only token ratios, and GlobalMMLU Culturally Sensitive vs. Agnostic splits in the individual responses.

1. Scope of the Pre-Training Comparisons and Public Resource Gap

FineWeb2-HQ-QA is a dataset paper. In practice, pre-training practitioners often select, combine, and refine existing public datasets, such as English-source QA translated into target languages (Nemotron TDQA) or QA generated from target-language web documents (FineWeb2-HQ-QA), and we believe FineWeb2-HQ-QA is a valuable addition to the open multilingual data landscape. While the Nemotron Nano 2 report describes target-language Wikipedia QA, nvidia/Nemotron-CC-v2 publicly released only the English (Diverse-QA) and translated (Translated-Diverse-QA) web subsets, not the multilingual Wikipedia QA subset. Other open datasets either target instruction tuning (e.g., UPDESH, MURI-IT) or synthesize reasoning traces from encyclopedic/open seeds (PleIAs/SYNTH). To the best of our knowledge, FineWeb2-HQ-QA is the first publicly available, large-scale (193.2B-token, 20-language) pre-training dataset of document-grounded QA generated directly from curated target-language web documents. Isolating the causal effect of translation versus source selection in downstream pre-training would require constructing matched translated counterparts (e.g., translating FineWeb2-HQ into English or English FineWeb-HQ into 20 languages), regenerating tens of billions of tokens, and running 50B-63B token cooldowns on those artificial intermediate datasets, which is beyond the scope of this work.

Instead, we evaluate the complete public resources under matched token budgets across two scales (3B, 8B) and two language mixtures, Balanced Mixture (BM) and Natural Mixture (NM), to document their practical trade-offs, and we characterize the differences between the two publicly available datasets directly at the data level (Sections 2-3).

2. New Experiment: 13-Language Wikidata Entity Analysis

Setup:

  • Sample: For all 13 BM languages, we randomly sampled 20,00020{,}000 document-QA samples per language from FineWeb2-HQ-QA (Native) and 20,00020{,}000 from Nemotron TDQA (TDQA), totaling 260,000260{,}000 samples each.
  • Entity Lexicon: For each language, we queried Wikidata for target-language names of entities from Home countries (where the language is official/native) vs. US/UK across five categories: countries, states/regions, cities (population ≥100k\ge 100\text{k}), major political parties, and well-known public figures (full names only), keeping only entities with Wikipedia articles in multiple languages. Any name shared between Home and US/UK (e.g., identical city names) was excluded.
  • Filtering Ambiguous Common Words: Because some city names are also everyday words (e.g., German Essen (food), French Tours (towers), or UK Bath), we excluded single-word names that appear at high everyday word frequencies (wordfreq) or frequently in lowercase.
  • Metric: For each dataset and language, we count mentions of Home (HH) and US/UK (CC) entities across the 20,00020{,}000 samples and report the Home Share H/(H+C)H / (H + C) (in %).

Results:

LanguageLexicon SizeNative Home Share (%)TDQA Home Share (%)
jpn_Jpan4,81168.0%7.6%
cmn_Hani10,15759.2%13.5%
deu_Latn4,95149.7%6.1%
fra_Latn5,33141.8%7.2%
ita_Latn4,45133.9%3.4%
spa_Latn5,36431.9%7.5%
nld_Latn3,95031.6%2.1%
por_Latn4,27329.4%4.2%
arb_Arab4,44528.5%14.8%
pol_Latn3,94721.1%0.5%
swe_Latn3,79819.7%0.5%
rus_Cyrl4,37217.4%1.2%
dan_Latn2,95612.4%0.4%

Across all 13 languages, FineWeb2-HQ-QA has a much higher share of Home-country entities than TDQA (12.4%-68.0% in Native vs. 0.4%-14.8% in TDQA). This holds, with a slightly larger average gap, when counting only geographic places (countries, regions, cities), and Native remains higher than TDQA in all 13 languages after excluding the 20 most frequent entities per language.

3. New Experiment: Two-Stage Human Evaluation

Setup:

  • Sample & Annotators: We randomly sampled 50 German (398398 QAs) and 50 French (400400 QAs) retained documents from FineWeb2-HQ-QA, plus 50 German documents from Nemotron TDQA. Annotator A (native German, fluent French) annotated all 150 documents. Annotator B (a native-German author) independently annotated a 20-document (159159 QA) German overlap to measure inter-annotator agreement (IAA).
  • Two-Stage Protocol: In Stage 1 (blind to the QAs), the annotator read only the source passage, rated source quality, and, for documents anchored in the target language's region, pre-registered up to 3 specific local facts. Only after Stage 1 is locked does Stage 2 reveal the generated QAs to be rated for local-fact preservation (fully / partially / not preserved) and our audit rubric criteria (faithfulness, answerability, duplicates, formatting, false premises, triviality, and naturalness). Source naturalness/translationese was rated in a separate pass over the same documents. The 50 TDQA documents were rated in a separate session (source passage only).

Table 1: Provenance, Naturalness, and Local Fact Preservation

MetricFineWeb2-HQ-QA DE (n=50n=50)FineWeb2-HQ-QA FR (n=50n=50)Nemotron TDQA DE (n=50n=50)
Target-community locally anchored docs22.0% (11/50, DACH)10.0% (5/50, Francophonie)0.0% (0/50 DACH)
**Pre-registered local facts in QAs: fully / partially / not preserved13 / 15 / 5 (of 33)9 / 5 / 1 (of 15)0 DACH facts in source
Source naturalness / translationese issue30.0% (15/50)40.0% (20/50)98.0% (49/50)
QA-level naturalness79.6% natural86.8% naturalN/A

Table 2: Human QA Audit, Double-Annotated IAA (n=159n=159 DE QAs), and Gemini 3.6 Flash Agreement (n=798n=798 QAs)

CriterionHuman DE (n=398n=398)Human FR (n=400n=400)Human Combined (n=798n=798)Gemini Combined (n=798n=798)Human-Gemini Agree (κ\kappa)Human-Human IAA (n=159n=159 DE)
Any serious defect9.3%6.0%7.6%7.1%97.0% (κ=0.78\kappa = 0.78)97.5% (κ=0.87\kappa = 0.87)
Serious unfaithful6.5%3.2%4.9%5.1%97.7% (κ=0.76\kappa = 0.76)96.2% (κ=0.77\kappa = 0.77)
Serious unanswerable4.8%4.0%4.4%3.3%97.6% (κ=0.68\kappa = 0.68)96.9% (κ=0.77\kappa = 0.77)
Serious other5.5%2.5%4.0%1.3%96.7% (κ=0.37\kappa = 0.37)97.5% (κ=0.70\kappa = 0.70)
Any issue (incl. minor)38.4%34.5%36.5%10.9%74.2% (κ=0.35\kappa = 0.35)81.1% (κ=0.60\kappa = 0.60)

Summary of Findings:

  1. Translation artifacts vs. natural phrasing: While some target-language web passages also exhibit translationese or low linguistic quality (30% DE, 40% FR), 98% (49/50) of German TDQA samples show translationese/unnatural phrasing. 83.2% of FineWeb2-HQ-QA QAs are rated natural.
  2. Local knowledge preservation: Of 48 pre-registered DACH and Francophonie facts, 22 (45.8%) are fully and 20 (41.7%) partially preserved in the generated QAs (87.5% fully or partially), and only 6 (12.5%) are lost. German TDQA had 0/50 DACH documents (instead containing translated local facts about Ontario/Canada, the UK, and the US).
  3. LLM judge validation: Humans find no serious defect in 92.4% of QAs. On serious defects, Gemini closely matches human labels (7.1% vs. 7.6%, κ=0.78\kappa = 0.78; two-annotator κ=0.87\kappa = 0.87; κ=0.83\kappa = 0.83 DE, κ=0.69\kappa = 0.69 FR). Standalone κ\kappa is lower on the rare Serious other pool (false premises, malformed, trivial; κ=0.37\kappa = 0.37, 96.7% agree) mainly because 14/24 human-only cases are already flagged by Gemini as serious unfaithful/unanswerable. Gemini flags far fewer minor issues than humans (6.5% vs. 22.1% unfaithful incl. minor), so the paper's automated audit rates reflect serious defects. Document-level factual-error flags do not match humans (κ=0.13\kappa = 0.13), so we will remove that estimate from Section 3.3 (and note it in Appendix F).

We are currently integrating all new analyses, tables, and references into the manuscript and will upload the revised PDF over the next days.

We have uploaded the revised PDF, with changes highlighted in blue. The new analyses are in Appendices E.4–E.5 (paired SEs, GlobalMMLU CS/CA), F.1 (human evaluation) and G (entity locality, provenance and translationese).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.