acceptodds
Under review as a conference paper at ICLR 2027

KletterMix: Climbing Toward High-Quality German Pretraining Data

Abstract

High-quality pretraining data is a central ingredient in modern language models, but German-language resources remain far less developed than their English coun- terparts: they are often smaller, less carefully curated, weakly documented, and rarely validated through controlled training experiments. We introduce KletterMix, a high-quality German corpus for language model pretraining and annealing, de- signed as reusable dataset artifact for the natural language processing and modeling community. KletterMix is built by translating a state-of-the-art English pretraining corpus into German while preserving document boundaries, metadata, source struc- ture, and topical diversity. This construction yields a German corpus with the scale and diversity of a modern pretraining dataset, while enabling direct comparison to its English source. We document the dataset through a broad set of corpus-level analyses, including translation quality, document length distributions, topic cov- erage, and source composition. A COMETKiwi-supervised proxy enables scalable filtering, bringing the corpus’s aggregate linguistic profile closer to a matched German-web reference without uniformly selecting syntactically simpler text. Beyond dataset construction, we evaluate KletterMix through controlled pretraining and annealing ablations against established German corpora, we show that models trained on KletterMix achieve measurable improvements on German-language downstream evaluations. These results demonstrate that carefully curated translated data can substantially strengthen the German pretraining data ecosystem.

Primary area:
datasets and benchmarks
Keywords:
Pretraining Data, Annealing Data, LLM, Multilingual Data, German Data

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8qAPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a new German language text corpus, KletterMix, translated from ClimbMix. The authors develop a pipeline of translation and quality annotation, and compare a 12B token subset of their corpus (725B tokens total) with 2 existing datasets, GermanWeb and FineWeb2-DE. They compare their corpus as both a pre-training dataset and an annealing dataset, and evaluate on 4 downstream evaluation benchmarks, finding that on average, KletterMix leads to higher evaluation scores.

Critical strengths
  • Bringing advanced AI technologies to non-native English speakers is an important goal
  • This paper highlights the significant engineering work required to create a large-scale dataset
Critical weaknesses
  • In terms of scientific novelty, this paper doesn't add much to the existing literature. Translation pipelines have been demonstrated in previous works (including the comparisons), with minor variations in the source English dataset possibly explaining the differences in performance
  • The authors should dig deeper to better understand why their method led to gains, and what caused those improvements. For example, how can these experiments be used in the future to lead to higher-quality non-english datasets? I feel that there is a lot of presentation of the numbers, but is missing a deeper analysis
Questions for authors
  • Is there an ablation study on each of the components of your proposed process? How can the reader know whether your translation pipeline led to downstream performance improvements, or whether those gains simply come from a better English source dataset? Generally speaking, what evidence is there to explain why KletterMix leads to better performance over FineWeb2-DE and GermanWeb? What are the differences in these datasets? What insights could someone take away from this paper to produce better datasets in the future?
  • Is there any plan to train a model on larger subsets? It would be interesting to see how adding more tokens scales downstream performance
  • As far as I can tell, the evaluations performed in this work are on datasets that were translated from English (MMLU, PIQA, HellaSwag, ARC-C), is that correct? If so, wouldn't it be more representative to evaluate on a German-native dataset? Otherwise, the issues with translation that you mention in the limitations (lines 461-465) sections are valid concerns about the evaluation datasets that you have, in addition to being concerns about your training dataset.
  • In the introduction, you argue that English source data has been mixture design, but it's not clear to me. Can German data not be mixed in the same way? Is there a fundamental reason why taking native german data and mixing it yourself is not an appropriate action?
  • On lines 320-321 you describe some suspicious outputs from translation, but it's unclear what actions were taken based on the discovery
  • In "document-preserving translation" (line 186) you describe a lot of work around preserving quality of long documents. Did you perform any validation demonstrating that the methods that you chose were successful?
  • Lines 218-220 mention that documents are ranked by a hash of document identifier. What constitutes a high ranking, making one document better than another? Do you just mean "sorted" rather than "ranked"?
6 comments

We thank the reviewer for asking what our experiments teach about German training mixtures.

Contribution (W1). We contribute a recipe for translating curated pretraining mixtures at scale, building on prior translation-based pretraining [11] with contextual long-document handling and efficient quality selection (Sec. 3). We deliberately combine ClimbMix’s high-quality English source mixture [1] with a strong translation model [2], and our downstream experiments demonstrate the resulting data’s utility for German LLM pretraining and annealing. Translation can introduce language-specific issues and artifacts, so we complement it with further quality filtering. To make this practical at corpus scale, we approximate the established COMETKiwi quality signal with a lightweight proxy. At a fixed training budget, the 0.60 cutoff further widens KletterMix’s German Core lead over the tested baselines, while human assessment also finds fewer errors among retained translations (details below). Alongside the recipe, we contribute one of the largest open German pretraining resources to the community, comprising 725B GPT-2 tokens with document-level source alignment. It is qualitatively assessed and quantitatively validated through matched 0.6B and 7B training experiments, with higher German Core scores than FineWeb2-DE and GermanWeb at both scales.

What we learned (W2). Our experiments show that a curated English mixture can remain useful for German model training after translation, with further quality selection improving its downstream utility. At a fixed 12B-token budget, the ≥0.60 cutoff raises German Core from 38.7% to 40.2%, ahead of FineWeb2-DE (38.3%) and GermanWeb (36.8%); ≥0.50 and ≥0.55 score 37.1% and 37.4% (Tables 1, 27). This sweep identifies the strongest tested cutoff through model training. The human assessment also finds fewer errors among retained translations (App. E), providing complementary support for the quality-selection approach. The practical lesson is to build on strong source curation and validate the translated, filtered corpus through downstream training.

Pipeline design and validation (Q1). Quality guides the construction from the outset. We choose ClimbMix because its curated mixture has strong English pretraining results [1, Figs. 1–2]. We evaluated multiple candidate LLMs for translation and selected Qwen3.5-397B-A17B, informed by its published WMT24++ performance [2]. Our precision tests also show that FP8 closely matches FP16 on English–German (App. A.1).

Longer inputs can reduce translation quality [10], and generation budgets must accommodate both the English source and the generated German text. We therefore adapt the procedure to the selected model using multi-sentence chunks, preceding German translations as context, and output budgets determined by source length (App. A.2). This provides broader document context than the individual-sentence translation in Wang et al. [11]. We then approximate the established COMETKiwi quality signal with an inexpensive corpus-wide proxy, whose utility we assess through filtering ablations and human evaluation.

The matched 0.6B and 7B runs test the complete recipe against education-filtered FineWeb2-DE [3] and GermanWeb’s mixture of crawl, FineWeb2, and synthetic data [4]. From the same FineWeb2-DE checkpoint, KletterMix annealing yields 39.4% German Core versus 37.6% for GermanWeb (Table 1). Thus, prior evidence and our targeted checks support the component choices, while the training experiments validate their ability to preserve the inputs quality and combined utility for German pretraining and annealing.

Larger subsets (Q2). The additional results support KletterMix’s utility for longer training and at larger model scales. Within one 7B schedule using 20% KletterMix, zero-shot German Core rises from 47.56% at 2.10B to 49.89% at 12.00B total annealing tokens (App. C.4, Table 26). As a large-scale proof of concept, we evaluated two 30B A3B MoEs, one of which included Klettermix in the annealing phase. The one with Klettermix scored 85.3 versus 80.7 on German evaluation suite. It also scored higher on INCLUDE-DE (61.2% vs. 59.7%) and German HellaSwag (74.0% vs. 66.6%) [5].

German evaluation (Q3). Our suite includes human-translated or post-edited Global-MMLU-Lite [7] and natively authored Global PIQA nonparallel-DE [8], as well as machine-translated HellaSwagX and ARCx [6] (App. A.5). Filtered 0.6B KletterMix ties FineWeb2-DE on native PIQA (Table 1). We additionally evaluated the held-out INCLUDE-DE benchmark set of native German regional exam questions [9]. At 7B and 20% German, KletterMix/FineWeb2-DE/GermanWeb score 38.13/35.25/33.09% (App. C.4, Table 23), extending the evidence to native German regional knowledge.

Mixture design (Q4). Yes. GermanWeb already blends web and synthetic data [4]. KletterMix adds a curated, source-aligned German component for such mixtures (Sec. 1).

Short translations (Q5). The short outputs in Fig. 2 motivated proxy filtering. We retain valid short outputs with their scores; all four low-quality examples in App. B.6.1 fall below the tested cutoffs. A target-to-source word ratio below 0.5 additionally flagged three retained pairs that at least two annotators rated as having extensive errors. This check would flag an estimated 0.42% of documents passing the 0.60 cutoff (App. E), illustrating how source alignment enables a complementary omission check.

Long documents (Q6). We added source–target checks stratified by document length on 58,055 aligned pairs (App. B.7, Table 11). For long sources (≥2,048 words) containing numbers, topic-balanced recovery of source numbers is 85.15% in retained translations versus 78.05% in rejected translations (+7.10 points; 95% interval [3.87, 10.34]). A separate full-text human assessment of 16 long-document pairs shows the same pattern: all three annotators flag a lower proportion of retained than rejected translations (App. E, Table 30). Our procedure uses chunks of up to 20k source tokens, a 2k-token previous-translation window, and a 32,768-token output cap (App. A.2) [2, 10]. Together, these automatic and human assessments support our translation and quality-selection procedure for long documents.

Hash order (Q7). Yes, “sorted” is clearer: we select the smallest seeded document-ID hashes per source-cluster quota (Sec. 3).

[1] Diao et al. Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training. 2025. NeurIPS.

[2] Qwen Team. Qwen3.5: Towards Native Multimodal Agents. 2026. Qwen technical blog and Hugging Face model card.

[3] Penedo et al. FineWeb2: One Pipeline to Scale Them All — Adapting Pre-Training Data Processing to Every Language. 2025. COLM.

[4] Burns et al. Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generation. 2026. EACL.

[5] Anonymous. Citation withheld for double-blind review.

[6] Thellmann et al. Towards Multilingual LLM Evaluation for European Languages. 2024. arXiv preprint, arXiv:2410.08928.

[7] Singh et al. Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation. 2025. ACL.

[8] Chang et al. Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures. 2025. arXiv preprint, arXiv:2510.24081.

[9] Romanou et al. INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge. 2025. ICLR.

[10] Peng et al. Investigating Length Issues in Document-level Machine Translation. 2025. Machine Translation Summit.

[11] Wang et al. Multilingual Language Model Pretraining using Machine-translated Data. 2025. EMNLP.

AuthorsOpenReview ↗

Open Questions?

Dear Reviewer 8qAP,

Thank you again for your constructive review and for emphasizing the importance of understanding what makes translated pretraining data useful. We have expanded our analyses and clarified the practical lessons of KletterMix, with changes marked in red-orange.

  • Ablations and insights. At a fixed 12B-token budget, filtering at the 0.60 cutoff improves German Core from 38.7% to 40.2% (Tables 1, 27). We now emphasize the importance of validating filtering thresholds through downstream training.
  • Translation fidelity. We added a three-annotator human audit and source–target checks stratified by document length. Retained translations show fewer flagged errors in the audit and better source-number recovery in long documents (Apps. B.7, E).
  • Native German evaluation. KletterMix achieves higher INCLUDE-DE point estimates than FineWeb2-DE and GermanWeb at both model scales (App. C.4).
  • Training budget and reproducibility. We report additional results along the 7B training schedule and clarify long-document handling, mixture design, and sampling (Apps. A.2, C.3–C.4).

As the discussion period is coming to a close, we would appreciate knowing whether these additions address your concerns. If anything remains unresolved, a brief indication would help us respond directly.

If your concerns are addressed, we would be grateful if you would consider updating your assessment.

Best regards,
The Authors

Reviewer_Sk4pOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

KletterMix translates ClimbMix into a reported 725B-token German pretraining corpus using Qwen3.5 and annotates it with a target-only proxy trained on COMETKiwi scores. Matched 0.6B pretraining and annealing runs compare German corpora, and a 7B study varies the KletterMix share of an annealing mixture. The best filtered 0.6B split reaches a German Core Avg. of 40.2, versus 38.3 for FineWeb2-DE. At 7B, a 15% share gives the highest German mean (53.2, versus 51.2 and 51.7 for the native controls). Gains are most consistent on HellaSwag and ARC-C; English performance remains broadly stable.

Critical strengths
  • A large, well-documented resource. The 725B-token corpus is built with a carefully engineered translation pipeline whose configuration is documented in detail.
  • Controlled training comparisons. Corpora are compared under matched recipes and budgets at two model scales, with native German controls.
  • Evaluation beyond downstream scores. The proxy validation and linguistic analysis give useful insight into what filtering does.
Critical weaknesses
  1. At 7B, the advantage over native German corpora rests mainly on one comparison. That comparison (15% share, 5-shot) uses KletterMix's own best share; in the other three matched settings (20% share or zero-shot), its lead over the stronger native corpus is only 0.02–0.42 points (Tables 17, 19, 21), from single runs with no paired test. The 0.6B results also favor KletterMix, but mainly on HellaSwag and ARC-C.
  2. The gains may partly reflect machine-translated benchmarks. They concentrate on HellaSwag and ARC-C, whose German versions are DeepL translations (Thellmann et al., 2024), so part of the gain may come from translated training data matching translated test sets rather than from "reasoning transfer".
  3. The comparisons do not isolate the proposed mechanism. Source, translation, filtering, and sampling all differ between corpora; in the 7B runs, KletterMix is quality-ranked while the native controls' sampling is unspecified. Since translated pretraining data is not new (Doshi et al., 2024; Wang et al., 2025), the main contribution is the German resource, its pipeline, and its evaluation, and the mechanism claim should be narrowed.
  4. Translation fidelity is not audited. Without source length, the target-only proxy cannot tell a truncated translation from a short complete one, the failure the paper flags for long documents. The main release applies only basic validity checks, which refusals and translated code would pass.
Questions for authors
  1. Can the authors report a source–target audit of retained and rejected translations (e.g., at the 0.60 threshold), stratified by source length, with failure rates for omissions, mistranslations, and added content? How were long documents segmented or truncated for COMETKiwi scoring?
  2. Which German MMLU and PIQA versions were used, and are they human-translated or natively written? If so, the absence of consistent gains on them would support the translated-benchmark explanation. Evaluating all 0.6B checkpoints on a common, deduplicated native-German held-out set would also test the advantage on native text without new training.
  3. If the reasoning-transfer claim is kept, which control supports it, e.g., the same sources translated by another method, or mixtures varied at matched quality and budget?
  4. How were the FineWeb2-DE and GermanWeb shares sampled in the 7B runs, and were they quality-ranked like KletterMix?
4 comments

We thank the reviewer for prompting a closer look at the matched comparisons, benchmark provenance, and translation quality.

Matched results across scales (W1). At 7B, we compare all three corpora at matched 15% and 20% German shares under both five-shot and zero-shot protocols. This includes the 20% setting, where both German controls achieve higher German Core scores than at 15%. The 0.6B experiments provide additional comparisons under matched 12B-token German pretraining: unfiltered KletterMix achieves a German Core score of 38.7 (40.2 with the 0.60 filter), versus 38.3 for FineWeb2-DE and 36.8 for GermanWeb. Continuing from the same FineWeb2-DE checkpoint yields 39.4 with KletterMix versus 37.6 with GermanWeb (Sec. 5, Table 1).

We have added paired item-bootstrap analyses [1] for all four matched 7B settings (App. C.4, Table 25). KletterMix has the highest German Core point estimate in each setting. The largest lead over the stronger German control is +1.51 points [−0.48, 3.48] at the 15% five-shot setting selected in the sweep. All four aggregate intervals include zero; these intervals quantify item uncertainty for the evaluated checkpoints. Together, these comparisons support KletterMix as a competitive German training corpus across the evaluated pretraining and annealing settings, with evidence extending across both model scales.

Benchmark provenance (W2, Q2). Global-MMLU-Lite-DE uses 400 human-translated or post-edited English-origin questions [2]. Global PIQA nonparallel-DE uses 100 questions written by native German speakers [3]. HellaSwagX and ARCx are machine-translated [4] (App. A.5). Additional zero-shot PIQA comparisons are reported in Table 24.

We extended this evaluation with the held-out INCLUDE-DE benchmark [5], comprising regional-exam questions originally authored in German. All eight existing 0.6B checkpoints were evaluated zero-shot on the same official German test items, without additional training (Table 24).

On INCLUDE-DE, unfiltered 0.6B KletterMix scores 30.22%, versus 27.34% for FineWeb2-DE and 19.42% for GermanWeb. At 7B and 20% German, the corresponding scores are 38.13%, 35.25%, and 33.09% (Table 23). The 7B difference relative to FineWeb2-DE is +2.88 points (paired 95% interval [−2.91, 8.63]).

KletterMix thus also leads both controls in INCLUDE point estimates at both scales, extending the evidence for its utility to native German regional knowledge.

Design choices and ablations (W3, Q3). Our construction combines components supported by prior work with targeted validation and ablations in this study. ClimbMix’s curated mixture was validated through English pretraining experiments [7], and Qwen3.5-397B-A17B was selected for its published WMT24++ translation performance [6]. We compare FP8 and FP16 translation quality (App. A.1) and validate our proxy against COMETKiwi [8] on 18,275 disjoint documents (App. A.4).

Our filtering ablation holds the source corpus and translation procedure fixed, comparing unfiltered KletterMix with the ≥0.50, ≥0.55 and ≥0.60 variants under a matched 12B-token training budget. The ≥0.60 cutoff raises German Core from 38.7% to 40.2% (Tables 1, 27). We additionally assess the resulting translations through source–target checks across document lengths and human evaluation (Apps. B.7, E).

The matched 0.6B and 7B training experiments then evaluate the resulting corpus in pretraining and annealing. At 7B and 20% German, KletterMix improves zero-shot German Core by 4.52 points [2.29, 6.76] over ClimbMix-only, with a paired 95% interval (Sec. 5). Sampling procedures are detailed in App. C.3 and below. These results support the revised claim that translating a curated English mixture, combined with validated quality selection, produces a useful German training resource (Sec. 6).

Source–target audit (W4, Q1). Thank you for highlighting the need for a direct fidelity audit. We added a human audit in which three bilingual annotators assessed 40 complete source–translation pairs across document lengths: 21 retained pairs (proxy ≥0.60) and 19 rejected pairs. Proxy scores and filtering assignments were hidden. In this selected panel, the annotators flagged at least one omission, addition, or mistranslation in 28.6–38.1% of retained pairs, compared with 68.4–78.9% of rejected pairs. These judgments support the proxy as a useful quality-selection signal. Tables 29–30 report results by error category and source length (App. E). The full-text human audit complements COMETKiwi, which scores each pair once using up to 510 content tokens from the start of each text, and the released proxy computes its document-level features over the full German translation (Apps. A.4, B.7). The released source alignment also enables a simple completeness check: a target-to-source word ratio below 0.5 identified three retained translations rated as having extensive errors by at least two annotators. This check would flag an estimated 0.42% of documents passing the 0.60 cutoff (App. E), illustrating how source alignment supports targeted checks alongside scalable proxy selection.

Sampling (Q4). No. We used seeded document-ID hashes for FineWeb2-DE and GermanWeb, with upstream education filtering for FineWeb2-DE. KletterMix used descending proxy score, so higher shares include lower-ranked documents. German budgets were 1.8B and 2.4B OLMo tokens, with fixed English shard positions (App. C.3).

[1] Dror et al. The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing. 2018. ACL.

[2] Singh et al. Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation. 2025. ACL.

[3] Chang et al. Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures. 2025. arXiv preprint, arXiv:2510.24081.

[4] Thellmann et al. Towards Multilingual LLM Evaluation for European Languages. 2024. arXiv preprint, arXiv:2410.08928.

[5] Romanou et al. INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge. 2025. ICLR.

[6] Qwen Team. Qwen3.5: Towards Native Multimodal Agents. 2026. Qwen technical blog and Hugging Face model card.

[7] Diao et al. Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training. 2025. NeurIPS.

[8] Rei et al. CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task. 2022. WMT.

AuthorsOpenReview ↗

Open Concerns?

Dear Reviewer Sk4p,

Thank you again for your thorough review and for recognizing the value of our corpus and controlled training comparisons. We have added further evaluations, fidelity analyses, and methodological clarifications, with changes marked in teal.

  • Matched comparisons. We added paired bootstrap analyses across all four 7B settings and revised our claims to reflect the uncertainty in aggregate differences (Table 25).
  • Native German evaluation. New INCLUDE-DE results extend the evaluation beyond translated benchmarks, with KletterMix achieving the highest point estimates at both model scales (App. C.4).
  • Translation fidelity. Three bilingual annotators assessed 40 complete source–translation pairs. Retained translations showed fewer flagged errors than rejected ones in this panel. We also added checks stratified by document length (Apps. B.7, E).
  • Sampling and interpretation. We clarified sampling for the native corpora and quality ranking for KletterMix, and focused the revised claims on the corpus’s empirical utility (App. C.3, Sec. 6).

With the discussion period nearing its end, we would appreciate knowing whether these additions address your concerns, or whether any specific issues remain.

If your concerns are addressed, we would be grateful if you would consider updating your assessment.

Best regards,
The Authors

Thank you for the authors’ reply and supplementary analysis. My questions have been completely resolved, and I have therefore increased my scores.

Reviewer_iyVrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a German large language model pretraining dataset by machine-translating the English ClimbMix dataset, resulting in 725B tokens that show good performance when compared to the German subset of FineWeb2 and GermanWeb datasets. Authors describe the pipeline consisting of length-aware routing, chunking, and dynamic output budgeting. Authors assess the quality of the translations using the reference-free COMETKiwi model, analyze the length distribution, and the data composition. Main evaluation is performed by training and evaluating 0.6B and 7B parameter models on downstream benchmarks, resulting in high aggregate benchmark performance compared to the tested baselines.

Critical strengths
  1. Paper contains experiments showing consistent improvement on ARC-C and HellaSwag benchmarks using their proposed dataset over the baseline FineWeb2 and GermanWeb datasets.
  2. Authors perform reference-free translation quality estimation on a pilot subset of the dataset using the COMETKiwi model. Authors also train a gradient-boosted proxy model on COMETKiwi scores to score the whole corpus and obtain the best result on the 0.6B parameter model using the filtered variant.
Critical weaknesses
  1. While the paper mentions that its limitation is that the dataset inherits the source biases from its English source data, which could potentially affect German cultural knowledge, the paper does not measure its effects. I think it would strengthen the paper to split the culturally sensitive and culturally agnostic parts of Global MMLU and include a comparison on a culturally-aware benchmark, such as INCLUDE [1].
  2. The benchmarks which the dataset improves on the most (ARC-C and HellaSwag) are translations of English benchmarks and may get an advantage from having the English source pool. This is a problem because it is not possible to tell whether the performance of the dataset comes from fitting the benchmark distribution better than other datasets.
  3. The main 725B token count and Figure 2 use the GPT-2 tokenizer that is English-centric and might inflate German token counts. While the paper comments the effect of the tokenizer on token counts in Section 4, it does not use a multilingual or German tokenizer or word counts to get an idea of the volume of the dataset.

[1] Romanou et al., “INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge”

Questions for authors
  1. Could you provide the Global MMLU evaluation split by the culturally agnostic and culturally sensitive subsets?
  2. Could you provide additional evaluation on benchmarks which were not sourced from English?
3 comments

We thank the reviewer for these suggestions. We have added the requested cultural breakdown, native-German evaluation, and alternative corpus volume estimates (App. C.4, Tables 23–24; App. D).

Cultural subsets (W1, Q1). Following your suggestion, we break down the matched 7B five-shot Global-MMLU-Lite results into the same 200 culturally sensitive (CS) and 200 culturally agnostic (CA) questions, using the benchmark’s labels [1].

German componentCS 5-shot (%)CA 5-shot (%)
15% FineWeb2-DE46.558.0
15% GermanWeb48.056.5
15% KletterMix49.556.5
---
20% FineWeb2-DE47.559.0
20% GermanWeb47.557.0
20% KletterMix47.557.5

KletterMix has the highest observed CS accuracy at 15% (49.5%) and ties both controls at 20% (47.5%). At 15%, the CS differences relative to FineWeb2-DE and GermanWeb are +3.0 points [−1.0, 7.0] and +1.5 points [−3.0, 6.0], respectively (paired 95% intervals). On CA, KletterMix is 1.5 points below FineWeb2-DE at both shares; it matches GermanWeb at 15% and is 0.5 points higher at 20%. Table 23 gives the matched results and per-cell standard errors. We additionally evaluated native German regional-exam knowledge with INCLUDE-DE [2], where KletterMix also has higher point estimates than both controls at both model scales (see below).

Taken together, the CS and INCLUDE results suggest that KletterMix is competitive with GermanWeb and FineWeb2-DE on the evaluated culturally sensitive and regional-knowledge tasks, with point estimates that match or favor KletterMix. This should motivate future work on whether complementing translated corpora with targeted German cultural and regional data can further improve performance on these tasks.

Evaluation beyond English-origin tasks (W2, Q2). Our original suite combines machine-translated HellaSwagX and ARCx [3], human-translated or post-edited English questions in Global-MMLU-Lite [1], and natively authored German questions in Global PIQA nonparallel (deu_latn) [4] (App. A.5).

On native PIQA, 0.60-filtered KletterMix at 0.6B scores 70% five-shot, tying FineWeb2-DE and exceeding GermanWeb’s 63% (Table 1). At 7B and 15% German, KletterMix scores 70% five-shot, versus 68% for FineWeb2-DE and 70% for GermanWeb (Table 21). Other share and shot settings show mixed PIQA results (Tables 18, 20–24).

We additionally evaluated INCLUDE-DE, which comprises native German regional exam questions [2]. At 0.6B, unfiltered KletterMix scores 30.22% zero-shot, versus 27.34% for FineWeb2-DE and 19.42% for GermanWeb (Table 24). The 7B zero-shot results are:

German shareFineWeb2-DE (%)GermanWeb (%)KletterMix (%)
15%33.8134.5335.97
20%35.2533.0938.13

At 20%, the differences are +2.88 points [−2.91, 8.63] versus FineWeb2-DE and +5.04 [0.00, 10.27] versus GermanWeb, with paired 95% intervals (Table 23).

Together, the competitive native PIQA performance and favorable INCLUDE point estimates at both scales support KletterMix’s utility beyond machine-translated benchmarks.

Corpus volume (W3). We use GPT-2 as a common reporting unit with the English ClimbMix release. The uncompressed JSONL release occupies about 2 TB and contains 553.3M documents and 1.882T Unicode code points. A stratified sample estimates 250.97B words [247.08B, 254.98B], counting runs of at least two Unicode letter-like characters. On the same sample, Aleph Alpha’s German–English Kolibri-1 tokenizer [5] gives an estimated 367B tokens [362B, 373B]. Brackets give 95% sampling intervals (App. D, Table 28).

References

[1] Singh et al. Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation. 2025. ACL.

[2] Romanou et al. INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge. 2025. ICLR.

[3] Thellmann et al. Towards Multilingual LLM Evaluation for European Languages. 2024. arXiv preprint, arXiv:2410.08928.

[4] Chang et al. Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures. 2025. arXiv preprint, arXiv:2510.24081.

[5] Aleph Alpha et al. Kolibri: A Sovereign European Model on the Pareto Frontier. 2026. arXiv preprint, arXiv:2610.31945.

AuthorsOpenReview ↗

Open Concerns?

Dear Reviewer iyVr,

Thank you again for your detailed review and constructive suggestions. We have added evaluations and clarifications addressing cultural knowledge, native German benchmarks, and corpus size, with changes marked in blue.

  • Cultural knowledge. We added the requested culturally sensitive/agnostic MMLU breakdown. KletterMix matches or exceeds both controls’ point estimates on the culturally sensitive subset at both German shares (App. C.4, Table 23).
  • Beyond translated benchmarks. We clarified benchmark provenance and added INCLUDE-DE, complementing the native German PIQA evaluation. KletterMix achieves the highest INCLUDE-DE point estimates at both model scales (Apps. A.5, C.4; Tables 23–24).
  • Corpus size. We now report approximately 251B words and 367B tokens using the German–English Kolibri-1 tokenizer, alongside the original GPT-2 count (App. D).

As the discussion period is coming to an end, we would appreciate knowing whether these additions address your concerns. If any issues remain, a brief pointer would help us respond.

If your concerns are addressed, we would be grateful if you would consider updating your assessment.

Best regards,
The Authors

Thank you for providing additional evaluations and computing the corpus volume estimates. Based on the CS split and INCLUDE-DE, W1 has been resolved as the results show no degradation on cultural knowledge. Regarding W2, the results seem to indicate that improved performance comes on translated benchmarks and is on-par for the rest as the intervals include zero, which should be transparently presented. Lastly, W3 regarding the corpus volume has been resolved now. Conditional on the presentation of W2 results, I increased my Rating score from 2 to 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We again thank the reviewers for their careful reading and constructive feedback. We have responded to all remaining concerns in the individual replies and revised the manuscript accordingly. To make the changes easy to locate, revisions are marked in blue for Reviewer iyVr, teal for Reviewer Sk4p, and red-orange for Reviewer 8qAP. We are looking forward to a fruitful discussion.

Bests, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.