- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a new German language text corpus, KletterMix, translated from ClimbMix. The authors develop a pipeline of translation and quality annotation, and compare a 12B token subset of their corpus (725B tokens total) with 2 existing datasets, GermanWeb and FineWeb2-DE. They compare their corpus as both a pre-training dataset and an annealing dataset, and evaluate on 4 downstream evaluation benchmarks, finding that on average, KletterMix leads to higher evaluation scores.
Critical strengths
- Bringing advanced AI technologies to non-native English speakers is an important goal
- This paper highlights the significant engineering work required to create a large-scale dataset
Critical weaknesses
- In terms of scientific novelty, this paper doesn't add much to the existing literature. Translation pipelines have been demonstrated in previous works (including the comparisons), with minor variations in the source English dataset possibly explaining the differences in performance
- The authors should dig deeper to better understand why their method led to gains, and what caused those improvements. For example, how can these experiments be used in the future to lead to higher-quality non-english datasets? I feel that there is a lot of presentation of the numbers, but is missing a deeper analysis
Questions for authors
- Is there an ablation study on each of the components of your proposed process? How can the reader know whether your translation pipeline led to downstream performance improvements, or whether those gains simply come from a better English source dataset? Generally speaking, what evidence is there to explain why KletterMix leads to better performance over FineWeb2-DE and GermanWeb? What are the differences in these datasets? What insights could someone take away from this paper to produce better datasets in the future?
- Is there any plan to train a model on larger subsets? It would be interesting to see how adding more tokens scales downstream performance
- As far as I can tell, the evaluations performed in this work are on datasets that were translated from English (MMLU, PIQA, HellaSwag, ARC-C), is that correct? If so, wouldn't it be more representative to evaluate on a German-native dataset? Otherwise, the issues with translation that you mention in the limitations (lines 461-465) sections are valid concerns about the evaluation datasets that you have, in addition to being concerns about your training dataset.
- In the introduction, you argue that English source data has been mixture design, but it's not clear to me. Can German data not be mixed in the same way? Is there a fundamental reason why taking native german data and mixing it yourself is not an appropriate action?
- On lines 320-321 you describe some suspicious outputs from translation, but it's unclear what actions were taken based on the discovery
- In "document-preserving translation" (line 186) you describe a lot of work around preserving quality of long documents. Did you perform any validation demonstrating that the methods that you chose were successful?
- Lines 218-220 mention that documents are ranked by a hash of document identifier. What constitutes a high ranking, making one document better than another? Do you just mean "sorted" rather than "ranked"?
6 comments
Part 2/2
Mixture design (Q4). Yes. GermanWeb already blends web and synthetic data [4]. KletterMix adds a curated, source-aligned German component for such mixtures (Sec. 1).
Short translations (Q5). The short outputs in Fig. 2 motivated proxy filtering. We retain valid short outputs with their scores; all four low-quality examples in App. B.6.1 fall below the tested cutoffs. A target-to-source word ratio below 0.5 additionally flagged three retained pairs that at least two annotators rated as having extensive errors. This check would flag an estimated 0.42% of documents passing the 0.60 cutoff (App. E), illustrating how source alignment enables a complementary omission check.
Long documents (Q6). We added source–target checks stratified by document length on 58,055 aligned pairs (App. B.7, Table 11). For long sources (≥2,048 words) containing numbers, topic-balanced recovery of source numbers is 85.15% in retained translations versus 78.05% in rejected translations (+7.10 points; 95% interval [3.87, 10.34]). A separate full-text human assessment of 16 long-document pairs shows the same pattern: all three annotators flag a lower proportion of retained than rejected translations (App. E, Table 30). Our procedure uses chunks of up to 20k source tokens, a 2k-token previous-translation window, and a 32,768-token output cap (App. A.2) [2, 10]. Together, these automatic and human assessments support our translation and quality-selection procedure for long documents.
Hash order (Q7). Yes, “sorted” is clearer: we select the smallest seeded document-ID hashes per source-cluster quota (Sec. 3).
[1] Diao et al. Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training. 2025. NeurIPS.
[2] Qwen Team. Qwen3.5: Towards Native Multimodal Agents. 2026. Qwen technical blog and Hugging Face model card.
[3] Penedo et al. FineWeb2: One Pipeline to Scale Them All — Adapting Pre-Training Data Processing to Every Language. 2025. COLM.
[4] Burns et al. Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generation. 2026. EACL.
[5] Anonymous. Citation withheld for double-blind review.
[6] Thellmann et al. Towards Multilingual LLM Evaluation for European Languages. 2024. arXiv preprint, arXiv:2410.08928.
[7] Singh et al. Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation. 2025. ACL.
[8] Chang et al. Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures. 2025. arXiv preprint, arXiv:2510.24081.
[9] Romanou et al. INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge. 2025. ICLR.
[10] Peng et al. Investigating Length Issues in Document-level Machine Translation. 2025. Machine Translation Summit.
[11] Wang et al. Multilingual Language Model Pretraining using Machine-translated Data. 2025. EMNLP.
Open Questions?
Dear Reviewer 8qAP,
Thank you again for your constructive review and for emphasizing the importance of understanding what makes translated pretraining data useful. We have expanded our analyses and clarified the practical lessons of KletterMix, with changes marked in red-orange.
- Ablations and insights. At a fixed 12B-token budget, filtering at the 0.60 cutoff improves German Core from 38.7% to 40.2% (Tables 1, 27). We now emphasize the importance of validating filtering thresholds through downstream training.
- Translation fidelity. We added a three-annotator human audit and source–target checks stratified by document length. Retained translations show fewer flagged errors in the audit and better source-number recovery in long documents (Apps. B.7, E).
- Native German evaluation. KletterMix achieves higher INCLUDE-DE point estimates than FineWeb2-DE and GermanWeb at both model scales (App. C.4).
- Training budget and reproducibility. We report additional results along the 7B training schedule and clarify long-document handling, mixture design, and sampling (Apps. A.2, C.3–C.4).
As the discussion period is coming to a close, we would appreciate knowing whether these additions address your concerns. If anything remains unresolved, a brief indication would help us respond directly.
If your concerns are addressed, we would be grateful if you would consider updating your assessment.
Best regards,
The Authors
Part 1/2
We thank the reviewer for asking what our experiments teach about German training mixtures.
Contribution (W1). We contribute a recipe for translating curated pretraining mixtures at scale, building on prior translation-based pretraining [11] with contextual long-document handling and efficient quality selection (Sec. 3). We deliberately combine ClimbMix’s high-quality English source mixture [1] with a strong translation model [2], and our downstream experiments demonstrate the resulting data’s utility for German LLM pretraining and annealing. Translation can introduce language-specific issues and artifacts, so we complement it with further quality filtering. To make this practical at corpus scale, we approximate the established COMETKiwi quality signal with a lightweight proxy. At a fixed training budget, the 0.60 cutoff further widens KletterMix’s German Core lead over the tested baselines, while human assessment also finds fewer errors among retained translations (details below). Alongside the recipe, we contribute one of the largest open German pretraining resources to the community, comprising 725B GPT-2 tokens with document-level source alignment. It is qualitatively assessed and quantitatively validated through matched 0.6B and 7B training experiments, with higher German Core scores than FineWeb2-DE and GermanWeb at both scales.
What we learned (W2). Our experiments show that a curated English mixture can remain useful for German model training after translation, with further quality selection improving its downstream utility. At a fixed 12B-token budget, the ≥0.60 cutoff raises German Core from 38.7% to 40.2%, ahead of FineWeb2-DE (38.3%) and GermanWeb (36.8%); ≥0.50 and ≥0.55 score 37.1% and 37.4% (Tables 1, 27). This sweep identifies the strongest tested cutoff through model training. The human assessment also finds fewer errors among retained translations (App. E), providing complementary support for the quality-selection approach. The practical lesson is to build on strong source curation and validate the translated, filtered corpus through downstream training.
Pipeline design and validation (Q1). Quality guides the construction from the outset. We choose ClimbMix because its curated mixture has strong English pretraining results [1, Figs. 1–2]. We evaluated multiple candidate LLMs for translation and selected Qwen3.5-397B-A17B, informed by its published WMT24++ performance [2]. Our precision tests also show that FP8 closely matches FP16 on English–German (App. A.1).
Longer inputs can reduce translation quality [10], and generation budgets must accommodate both the English source and the generated German text. We therefore adapt the procedure to the selected model using multi-sentence chunks, preceding German translations as context, and output budgets determined by source length (App. A.2). This provides broader document context than the individual-sentence translation in Wang et al. [11]. We then approximate the established COMETKiwi quality signal with an inexpensive corpus-wide proxy, whose utility we assess through filtering ablations and human evaluation.
The matched 0.6B and 7B runs test the complete recipe against education-filtered FineWeb2-DE [3] and GermanWeb’s mixture of crawl, FineWeb2, and synthetic data [4]. From the same FineWeb2-DE checkpoint, KletterMix annealing yields 39.4% German Core versus 37.6% for GermanWeb (Table 1). Thus, prior evidence and our targeted checks support the component choices, while the training experiments validate their ability to preserve the inputs quality and combined utility for German pretraining and annealing.
Larger subsets (Q2). The additional results support KletterMix’s utility for longer training and at larger model scales. Within one 7B schedule using 20% KletterMix, zero-shot German Core rises from 47.56% at 2.10B to 49.89% at 12.00B total annealing tokens (App. C.4, Table 26). As a large-scale proof of concept, we evaluated two 30B A3B MoEs, one of which included Klettermix in the annealing phase. The one with Klettermix scored 85.3 versus 80.7 on German evaluation suite. It also scored higher on INCLUDE-DE (61.2% vs. 59.7%) and German HellaSwag (74.0% vs. 66.6%) [5].
German evaluation (Q3). Our suite includes human-translated or post-edited Global-MMLU-Lite [7] and natively authored Global PIQA nonparallel-DE [8], as well as machine-translated HellaSwagX and ARCx [6] (App. A.5). Filtered 0.6B KletterMix ties FineWeb2-DE on native PIQA (Table 1). We additionally evaluated the held-out INCLUDE-DE benchmark set of native German regional exam questions [9]. At 7B and 20% German, KletterMix/FineWeb2-DE/GermanWeb score 38.13/35.25/33.09% (App. C.4, Table 23), extending the evidence to native German regional knowledge.