- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces a multilingual pre-training resource built by generating questions and answers directly from target-language web documents, called FineWeb2-HQ-QA. The authors evaluate whether this document-grounded QA provides useful training signal compared with the original documents, English-source QA, and translated QA. The results suggest that native QA is a useful complement to existing resources, with different trade-offs across knowledge and general language understanding rather than consistent superiority over translated data.
Critical strengths
- The paper contributes a useful alternative to English-centered synthetic data. Importantly, the design choice to generate QA pairs from target-language documents allows the resource to draw on content that may be less visible in English-source corpora. The value of the dataset is also shown by downstream training experiments.
- The paper is useful as a study of complementary multilingual data sources. The authors show that the translated QA pairs remains stronger in some settings and that combining resources does not always improve performance.
Critical weaknesses
- The connection between target-language sourcing and locally relevant knowledge is confusing. While the motivation of the paper suggests that this approach can recover content overlooked by English-centered resources, the actual experiments rather show downstream usefulness than they establish what distinctive knowledge the dataset contributes. The paper lacks a closer analysis of the content coverage, which makes it difficult to tell whether the dataset reflect locally specific information.
- The qualitative assessment of the dataset also is lacking. Although the conducted automated filtering provide useful checks, the QA is still generated by a single model. This is concerning since it may introduce generation that favor familiar facts, standardized language, and conventional question types.
Questions for authors
- How did the authors ensure or measure what is lost or changed when native documents are converted into synthetic QA? For example, have the authors looked into whether generation preserves culturally specific content and linguistic variation?
- Will users be able to trace generated QA back to its source and inspect why a sample was retained or removed?
- How did the authors distinguish useful QA augmentation from repeatedly restating information already present in the source?
- How can we ensure that the automated quality judgments reflect readers’ judgments in the target languages?
3 comments
Response to the Authors
Thank you for the response and for clarifying the scope of the comparison! I see FineWeb2-HQ-QA as an useful multilingual resource (especially playing a role as an alternative to more of English-centered synthetic data), and the experiments show that it can improve knowledge scores over the matched document-only baseline. My main concern remains that the comparison with traslated QA cannot isolate the effect of generating QA directly in the target language, since the source corpora and processing pipeline also differ. (Although the authors mention that running a fully controlled comparison is costly, I feel this is one of the most critical piece in the work to claim the contribution that it's making.) The revised framing makes that distinction clearer, but it does not change my overall assessment. I'll retain my original score.
Thank you for the constructive discussion and for recognizing FineWeb2-HQ-QA as a useful multilingual resource. We agree that our comparison with Nemotron TDQA does not isolate the effect of target-language generation, as noted explicitly in Section 1. As a dataset paper, our goal is to release an open 193B-token, 20-language pre-training resource, show consistent gains over the matched document-only baseline and mixing trade-offs across four 3B/8B settings, and (added in the revised PDF) characterize its target-region coverage, local-fact preservation, and translationese compared with TDQA (Appendix G), as well as its human-validated quality (Appendix F).
We thank the reviewer for the constructive questions on local content coverage, traceability, QA augmentation vs. restatement, and human validation of the automated audit. Please see the General Comment for the experimental setup and results of our 13-language Wikidata entity analysis (Section 2) and two-stage human evaluation (Section 3). Below we address your four questions directly.
Q1. Measuring What Is Preserved or Lost When Converting Native Documents to QA (W1 & Q1)
We measured local content coverage and preservation at two levels (General Comment Sections 2-3):
Corpus-wide locality across 13 languages: Across 260,000 samples in each dataset, FineWeb2-HQ-QA has a much higher share of target-country over US/UK entity mentions than TDQA in all 13 BM languages (e.g., 49.7% vs. 6.1% in German, 41.8% vs. 7.2% in French, 68.0% vs. 7.6% in Japanese).
Fact preservation and naturalness in QA generation: In Stage 1 of our human evaluation (blind to QAs), the annotator pre-registered 48 local facts across the locally anchored German (22%, DACH) and French (10%, Francophonie) documents. Stage 2 shows that 87.5% (42/48) of these facts are fully (22/48) or partially (20/48) preserved in the generated QAs. Overall, 83.2% of all 798 QAs were rated natural. While 30%-40% of source web documents in German and French showed translationese or non-fluent phrasing, this compares to 98% (49/50) with translationese artifacts in German TDQA.
Q2. Traceability to Source Documents and Filter Inspection (Q2)
datatrove4-stage filtering pipeline (completed</think>check, GlotLID, Gopher repetition, and 13-gram decontamination). Users can trace every QA pair back to its source passage, deterministically reproduce the 174.0M-document filtered dataset (and inspect which stage removed any sample), or customize per-language filter thresholds.Q3. Distinguishing Useful QA Augmentation from Restating Source Information (Q3)
Q4. Target-Language Quality Judgments & Single-Model Generation (W2 & Q4)
Serious other(κ=0.37, 96.7% agreement), 14 of the 24 human-only cases are already caught by Gemini as seriousunfaithfulorunanswerablerather than double-tagged. Across all 6 audited languages, repeat-run agreement is high (κ=0.76-0.84) and defect-injection recall is 99.5% (1,075/1,080). Because Gemini under-flags minor issues and its document-level factual-error flags do not match humans (κ=0.13), we will present the judge as validated for serious QA-level defects only (Section 3.3, Appendix F).Qwen3-30B-A3B): We usedQwen3-30B-A3B, the same model used to translate Nemotron DQA into TDQA (Section 3.1). While it preserves 87.5% (42/48) of pre-registered local facts (fully or partially) and produces 83.2% natural QAs across six prompted question formats, prompting a single model for "clear and concise" QAs favors standard written register over dialectal variation and can standardize phrasing. We will note this in Limitations (Section 6 / Appendix A).