acceptodds
Under review as a conference paper at ICLR 2027

Haidass: Bilingual Small Language Models Based on Data Curation and Language Scheduling

Abstract

Small language models (SLMs) offer clear advantages in efficiency and deployment, especially for on-device bilingual applications, but at the sub-200M scale, they remain English-centric: robust bilingual SLMs are scarce. Introducing a second language at this scale forces both languages to share a narrow parameter budget. To inform the design of such a bilingual SLM, we study how Chinese-English pretraining corpora interact. We quantify the cross-lingual interference between the two languages and how it depends on the mixing ratio, and compare several data strategies. Guided by this, we curate a balanced Chinese-English corpus and train Haidass-143M on 391.1B tokens, the first bilingual SLM at this scale, with a multi-stage real-to-synthetic data curation and language scheduling. It ranks 5th on the Open SLM Leaderboard. Building on this model, we post-train Haidass-Translate-143M. With direction-specific beam decoding, it achieves 30.36 BLEU and 21.40 chrF++ on FLORES-200 (enzh), outperforming the dedicated NLLB-200-distilled-600M by 7.92 BLEU and 4.66 chrF++, and essentially matches Qwen3-0.6B (30.94 BLEU, 21.10 chrF++) despite roughly 4 fewer parameters. This work provides empirical guidance for the training recipe design of SLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Small language model, Data curation, Language scheduling, Bilingual training recipe

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZkE6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Haidass-143M, a Chinese–English bilingual language model trained on 391B tokens using a multi-stage natural-to-synthetic data curriculum. The authors investigate quality filtering, language mixing, and cross-lingual interference in small language models. Haidass achieves competitive performance among similarly sized models, while its translation variant approaches substantially larger models on FLORES benchmarks.

Critical strengths
  1. Relevant research problem: The paper investigates Chinese–English bilingual language modeling under tight parameter constraints (sub-200M), a relatively underexplored setting with practical relevance for resource-constrained deployment.
  2. Useful empirical analysis: Provides controlled experiments quantifying the effects of data filtering, language allocation, and scheduling across small model scales, highlighting task-dependent trade-offs and the benefits of continuous language interleaving.
Critical weaknesses
  1. Limited novelty and insufficient research contribution: The paper mainly combines established techniques without introducing a sufficiently novel methodology or research contribution for ICLR. Its empirical findings largely confirm known observations, with limited new scientific insights.
  2. Missing ablation of the main contribution: The proposed natural-to-synthetic curriculum is not compared against fixed-mixture or alternative scheduling strategies, making its actual contribution unclear.
  3. Limited model performance: The model scores near random chance on several bilingual benchmarks, providing little evidence of meaningful bilingual text-generation capabilities. Translation gains are largely limited to EN→ZH, while ZH→EN substantially underperforms NLLB. Moreover, Haidass uses tuned beam search while baselines use greedy decoding, raising fairness concerns.
Questions for authors

Given the limited capabilities of this 143M-parameter model, could the authors clarify the motivation for developing Haidass, its intended practical use cases, and its advantages in real-world deployment?

7 comments

We sincerely thank the reviewer for the careful assessment of the scientific contribution, curriculum evidence, and practical value of Haidass. These comments have helped us clarify how the controlled experiments inform compact Chinese--English training. In the responses below, we relate the contribution to prior research on pretraining data, present the full-budget and initial-source comparisons, and explain what the bilingual and translation evaluations establish.

The experiments support several concrete findings. Additional quality filtering improves mean accuracy across five model sizes while producing task-dependent losses; interleaving preserves balanced language performance better than the tested block schedules at matched per-language exposure; and the original web recipe gives the highest four-task mean among the tested initial sources. At the reported 391.1B-token budget, the complete Haidass recipe improves accuracy on five of eight tasks relative to the fixed-mixture reference. The capability evaluations further show a Chinese reading-comprehension and text-likelihood advantage among the similarly sized references, together with competitive task-specific translation after post-training. We explain the scope and practical implications of each result below and for more detailed responses, please refer to the supplementary material Reply to Reviewer ZkE6.pdf

Comment 1: Limited novelty and insufficient research contribution: The paper mainly combines established techniques without introducing a sufficiently novel methodology or research contribution for ICLR. Its empirical findings largely confirm known observations, with limited new scientific insights.

Response: We appreciate the reviewer's request for a more precise account of the research contribution. Our study examines how data selection and language exposure affect the capabilities retained by a compact Chinese--English model. We use an established decoder-only Transformer backbone so that the controlled studies can examine these data decisions under matched architectural and training settings within each model size.

Pretraining-data design is an established research problem with contributions grounded in controlled empirical analysis. DoReMi learns domain weights through a smaller proxy model, while RegMix predicts mixture performance from many small training runs . SmolLM2 combines dataset ablations with a documented small-model training recipe. Within this research direction, Haidass focuses on two questions: whether additional filtering benefits the same capabilities across compact model scales, and whether equal cumulative language exposure produces equal bilingual retention when its temporal order changes.

Section 4.1, Table 5, Figure 3, and Appendix C compare the full Ultra-FineWeb-en pool with its additional HQ subset, defined by fastText quality scores of at least 0.9. Both conditions therefore start from an already filtered web source. At each of five model sizes, from approximately 56M to 803M parameters, the comparison uses the same architecture, training hyperparameters, and approximately 20B training tokens. HQ improves the final ten-task mean at every evaluated size under both raw and length-normalized candidate scoring, by 0.39--1.06 and 0.75--1.23 percentage points, respectively. However, the gains are neither uniform across tasks nor monotonic in model size. At 469M, raw accuracy improves by 6.44 points on ARC-Easy and 3.67 on ARC-Challenge, while decreasing by 1.11 on HellaSwag and 0.54 on PIQA.

The resulting insight concerns what an aggregate improvement can conceal. A selection rule that improves the average can reduce particular capabilities, so its usefulness depends on the intended task profile. The cross-scale and task-level results provide a concrete basis for evaluating filtering through its capability trade-offs and for retaining complementary sources when designing a mixture. They also avoid assuming that a stronger filter or a larger model necessarily produces a larger benefit.

Prior multilingual research shows that language-modeling outcomes depend on data availability and model size. Our ordering experiment examines whether changing when each language is presented alters the final outcome when both languages receive the same cumulative token totals. Section 4.2, Figure 4(b), and Appendix D compare sequence-level IID interleaving, eight alternating monolingual blocks, and a single language switch in either direction. The 43M and 143M models use 100M and 331M training tokens, respectively, with a 50:50 allocation held fixed within each size. Eight-block alternation increases the equal-weight mean of the two languages' relative held-out losses by 4.57% and 4.62% compared with IID. These changes use byte-normalized packed-block loss and the corresponding same-size IID reference, as defined in Appendix D, Eq. (3).

The single-switch results reveal a larger and asymmetric retention cost. At 143M, English-then-Chinese training increases English loss by 38.22% relative to IID, whereas the reverse order increases Chinese loss by 74.02%. In both directions, the language encountered first suffers the larger deterioration despite receiving the same cumulative exposure as under IID. This distinguishes the ordering effect from the separate language-ratio experiment, in which changing the ratio necessarily changes the training tokens allocated to each language.

These experiments add quantitative guidance for the studied compact bilingual setting: task-level performance is needed to interpret filtering gains, and final language-token totals are insufficient to characterize retention. The evidence supports assessing data selection against the intended capabilities and maintaining interleaved exposure when balanced language retention is required. The released model and documented recipe provide a concrete setting for applying and further examining these findings, with task-specific adaptation demonstrating their practical relevance.

Comment 2: Missing ablation of the main contribution: The proposed natural-to-synthetic curriculum is not compared against fixed-mixture or alternative scheduling strategies, making its actual contribution unclear.

Response: Thank you for requesting stronger experimental support for the training design. We report two complementary comparisons: a fixed-mixture reference at the released model's total token budget and a comparison of the initial data sources under matched training settings. The recipe progressively introduces rewritten and question--answer text while retaining web text; its motivation is to combine an informative initial source with additional ways of presenting knowledge within the same next-token prediction objective.

The fixed-mixture reference was trained for 391.1B processed tokens. At this full budget, the Haidass recipe achieves higher raw accuracy on five of the eight evaluated English tasks: ARC-Easy, ARC-Challenge, HellaSwag, PIQA, and ArithMark. The respective improvements are 1.81, 0.94, 1.43, 1.47, and 1.90 percentage points. The same five tasks also improve under length-normalized scoring, so the observed advantage is retained under both evaluation procedures.

Across all eight tasks, the unweighted mean raw accuracy increases from 38.62% for the fixed mixture to 39.42% for our curriculum-based recipe, an improvement of 0.80 percentage points. Mean length-normalized accuracy increases from 41.01% to 41.87%, an improvement of 0.86 points. Although OpenBookQA, WinoGrande, and AGIEval do not show uniform improvements, the overall results favor the proposed recipe under both scoring procedures.

These findings extend the evidence for our training design from controlled component studies to the full pretraining budget. The smaller-budget experiments show that quality filtering improves aggregate performance while introducing task-dependent trade-offs, and that continuous language interleaving better preserves balanced bilingual performance under matched per-language exposure. The full-budget comparison further shows that the complete recipe incorporating these principles achieves measurable aggregate gains over the tested fixed-mixture reference. Controlled experiments thus motivate the individual data choices, while the full-budget evaluation demonstrates improved aggregate performance when they are integrated into the proposed multi-stage recipe at the target training scale.

We also train three 143M models from random initialization on the original Ultra-en web recipe, L3 multi-style rewrites, or L3 question--answer data for approximately 20B tokens each. They share the architecture, tokenizer, sequence length, global training batch size, learning-rate configuration, and training seed. The original web recipe achieves the highest four-task mean under both scoring procedures: 44.26% raw accuracy and 43.95% normalized accuracy, compared with 41.42%/41.44% for L3 multi-style and 42.24%/42.07% for L3 QA. Its normalized mean also exceeds both alternatives at the shared checkpoints of approximately 5.24B, 10.49B, and 15.73B tokens. The task-level results retain complementary behavior: both L3 recipes improve normalized ARC-Challenge, and L3 QA slightly improves raw PIQA.

This experiment provides direct support for the initial web-source choice among the tested recipes. It compares sources from initialization, rather than exchanging their order at identical cumulative source exposure, and therefore establishes early-learning behavior without determining the optimal transition time for later data. The subsequent use of rewritten text has a separate rationale: WRAP shows that combining web documents with synthetic rephrasings can improve pretraining efficiency and downstream performance in its evaluated settings . In Haidass, multi-style and question--answer sources provide alternative presentations of information while preserving the next-token prediction objective described in Section-2.2.

Smaller controlled studies examine individual choices at the evaluated scale. Data Mixing Laws shows that mixture rankings can change with model size and training duration . We therefore use the component experiments to characterize filtering, exposure order, and initial-source effects, and assess the complete recipe separately at 391.1B reported tokens. A schedule-only comparison would additionally hold the tokenizer, source data, and cumulative source exposure fixed. Together, the results support an initial web foundation, continued bilingual exposure, and complementary text forms: the original web recipe gives the best average early performance among the tested sources, interleaving favors balanced retention at matched exposure, and the complete recipe improves five evaluations at the full reported budget.

Reviewer_EUTjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces a compact language model named Haidass, which has only 143M parameters and ranked 5th on the open SLM leaderboard. Authors illustrate the 6-stage training pipeline and progressively train this compact language model with 391B tokens. Experimental results on the open SLM benchmark and translation dataset report the performance of Haidass-143M. Further analyses show that the importance of data quality and bilingual samples mixing policy.

Critical strengths
  • The public model weight and training pipeline may be useful for the research community.
Critical weaknesses
  • This paper is a technical report rather than a sound research paper. For example, the most of contents are documenting the training pipeline of this model, and the performance of this model, lacking ablation studies and fair baseline methods to show the effectiveness of designed training policy.

  • Lacking fair baseline methods in Section 3.3. The model trained totally for EN-ZH bilingual translation, while the other models compared are general language models like Qwen2.5-0.5B-Instruct and multilingual translation model like NLLB-200-distilled-600M. To ensure a fair comparison, it should be compared with models trained using the same three-stage training strategy.

  • Most of training policies are introduced from the authors and not supported by the experimental results. For example, why is this model first trained on English HQ, then introduced DCLM? Why the percentage of Chinese corpus is increased first, then decreased to 18%?

Questions for authors

Refer to weakness part:

  • Why was the model first trained on English HQ before subsequently incorporating DCLM?

  • Why did the proportion of Chinese-language corpora first increase, subsequently drop to 18%?

  • Why was the model first trained on web corpus before synthetic text? Are there any experiments to support this strategy?

5 comments

We sincerely thank the reviewer for the careful assessment of our experimental evidence and the fairness of the comparisons. Your comments have helped us clarify the research contribution and strengthen the connection between the training design and its empirical support.

The revised evidence includes controlled quality-filtering experiments across five model sizes, language-order experiments with matched per-language exposure, and a completed fixed-mixture comparison at the released model's 391.1B-token budget. At this full budget, our recipe improves the eight-task mean by 0.80 percentage points under raw scoring and 0.86 points under length-normalized scoring. The component studies show consistent aggregate benefits from quality filtering and better balanced bilingual retention with continuous language interleaving.

We have also applied the same three-stage translation adaptation to three similarly sized pretrained baselines. Haidass achieves the strongest English-to-Chinese results among these models and remains close to the best Chinese-to-English baseline. A separate initialization control quantifies the benefit of Haidass pretraining under a shared translation SFT setup.

Finally, we provide a controlled comparison of initial data sources and clarify the rationale for introducing DCLM, adjusting Chinese web sampling weights, and incorporating rewritten and question-answer data. We distinguish the choices directly evaluated by these experiments from the specific timing and weights used in the complete recipe. We address each concern below. and for more detailed responses, please refer to the supplementary material Reply to Reviewer EUTj.pdf

Comment 1:

This paper is a technical report rather than a sound research paper. For example, the most of contents are documenting the training pipeline of this model, and the performance of this model, lacking ablation studies and fair baseline methods to show the effectiveness of designed training policy.

Response:

We sincerely thank the reviewer for examining the experimental support and comparison fairness. Our research question concerns how data selection and sustained bilingual exposure affect learning in a compact Chinese--English model. Controlled studies, a completed full-budget comparison, and the common translation-adaptation comparison below now connect the training choices to empirical evidence.

Section 4.1 and Appendix C compare the full Ultra-FineWeb-en pool with its additional high-quality subset, defined by a fastText score of at least 0.90.9. Within each of five model sizes, architecture, training hyperparameters, and approximately 20B training tokens are matched. HQ improves the final ten-task mean at every size: by 0.39--1.06 percentage points in raw accuracy and 0.75--1.23 points in length-normalized accuracy (manuscript Table 5). ARC gains coexist with HellaSwag and PIQA losses, supporting improved aggregate performance while motivating complementary sources to retain capability coverage.

At the released model's full budget, the fixed-mixture reference was trained for 746,000 updates, or 391.1B reported tokens. Both final checkpoints use the same zero-shot evaluator, BF16 precision, batch size 16, and no chat template. Across eight tasks, our recipe increases mean raw accuracy from 38.62% to 39.42% and mean length-normalized accuracy from 41.01% to 41.87%, gains of 0.80 and 0.86 percentage points. Raw accuracy improves on ARC-Easy, ARC-Challenge, HellaSwag, PIQA, and ArithMark by 1.81, 0.94, 1.43, 1.47, and 1.90 points; the same five tasks improve under normalized scoring. OpenBookQA, WinoGrande, and AGIEval do not improve uniformly.

The matched-exposure experiment below additionally tests language ordering. Together, these studies provide complementary evidence: component controls motivate filtering and continuous interleaving, while the full-budget comparison shows aggregate gains for their integration into the complete recipe over the tested fixed mixture.

Question: How can the effects of cross-lingual interference be distinguished from differences in per-language training exposure?

Response:

We agree that these factors require separate controls. In the language-ratio study, a fixed total budget NN gives Nzh=rNN_{\mathrm{zh}}=rN and Nen=(1−r)NN_{\mathrm{en}}=(1-r)N for Chinese share rr. Varying rr changes both per-language exposure and the joint training distribution, so bilingual--monolingual perplexity gaps measure a combined allocation trade-off and cannot be attributed solely to interference.

The language-order experiment instead fixes total English and Chinese tokens at a 50:5050{:}50 allocation and varies their order: sequence-level IID interleaving, eight alternating monolingual blocks, or one language switch in either direction. All schedules use 331M tokens at 143M parameters and 100M tokens at 43M parameters. The balanced loss penalty is Ps=50∑ℓ∈{en,zh}(L~ℓ,s/L~ℓ,IID−1)P_s=50\sum_{\ell\in\{\mathrm{en},\mathrm{zh}\}}(\widetilde L_{\ell,s}/\widetilde L_{\ell,\mathrm{IID}}-1), where L~\widetilde L denotes held-out byte-normalized packed-block loss (Appendix D, Eq. (3)).

IID interleaving gives the lowest balanced loss at both scales. At 143M, eight-block alternation incurs a 4.62% penalty. English-then-Chinese raises English and Chinese losses by 38.22% and 1.25%; Chinese-then-English raises Chinese loss by 74.02% and changes English loss by -1.21%. The first language therefore deteriorates most despite matched cumulative exposure. Its loss increases from its training minimum to the final checkpoint by 0.46 for English and 1.10 for Chinese under the respective single-switch schedules, providing additional evidence of forgetting and recency effects.

Thus, the ratio study informs allocation, while the ordering study directly supports continuous bilingual exposure over the tested block schedules. This distinction grounds our scheduling choice in improved balanced retention without treating every perplexity gap as pure interference (Section 4.2, Figure 4, and Appendices D--E).

Comment 2:

Lacking fair baseline methods in Section 3.3. The model trained totally for EN-ZH bilingual translation, while the other models compared are general language models like Qwen2.5-0.5B-Instruct and multilingual translation model like NLLB-200-distilled-600M. To ensure a fair comparison, it should be compared with models trained using the same three-stage training strategy.

Response:

We thank the reviewer for suggesting this comparison. We have added three similarly sized pretrained baselines, SmolLM2-135M, Cagliostro-v3-146M, and MobileLLM-140M, and applied the same three-stage adaptation procedure used for Haidass-143M: supervised translation fine-tuning, annealing on a curated data mixture, and sequence-level distillation. This matches the downstream adaptation strategy; it does not establish equal upstream pretraining data or compute budgets. The final adapted Haidass checkpoint is Haidass-Translate-143M.

All four models are evaluated on FLORES-200 dev with greedy decoding. Reporting BLEU/chrF++ for EN$\rightarrow$ZH and ZH$\rightarrow$EN, respectively, the results are SmolLM2: 23.36/18.22 and 18.01/44.47; Cagliostro: 19.88/15.81 and 16.13/42.05; MobileLLM: 26.02/18.57 and 15.11/42.07; and Haidass: 28.32/19.49 and 17.72/44.32. Haidass ranks first on both English-to-Chinese metrics, exceeding MobileLLM by 2.30 BLEU and 0.92 chrF++. It ranks second on both Chinese-to-English metrics, only 0.29 BLEU and 0.15 chrF++ below SmolLM2. Thus, its advantage for translation into Chinese persists when alternative compact pretrained models receive the same adaptation strategy, while reverse-direction performance remains competitive.

A separate initialization control compares random initialization with Haidass pretraining using the same Haidass architecture, translation data, and two-epoch SFT setup on FLORES-200 dev. These are two-epoch SFT checkpoints, distinct from the final three-stage models above. For EN$\rightarrow$ZH, BLEU/chrF++ rises from 12.0411/9.4333 to 23.8931/17.5480, gains of 11.85/8.11 points; for ZH$\rightarrow$EN, it rises from 5.4718/27.3139 to 14.6489/40.7619, gains of 9.18/13.45 points. This demonstrates the benefit of the pretrained starting point within a shared downstream SFT budget.

The larger reference-model comparison remains separate in Section 3.3, using common greedy decoding on FLORES+ devtest. Haidass achieves 29.23 BLEU for English-to-Chinese and 17.50 for Chinese-to-English, trailing Qwen2.5-0.5B-Instruct by 0.09 and 0.54 BLEU, respectively. Relative to NLLB-200-distilled-600M, it scores 6.16 BLEU higher for English-to-Chinese and 6.80 lower for Chinese-to-English. These models have different adaptation histories, so their results provide broader performance context; the added compact-model experiment addresses the requested comparison under a common adaptation strategy.

Together, the controls support Haidass-143M as an effective pretrained basis for bidirectional translation: pretraining substantially improves both directions over random initialization, and common three-stage adaptation yields the strongest English-to-Chinese results among the compact models tested while remaining close to the best Chinese-to-English baseline. We have added the comparison to Section 3.3 and clarified its relationship to the initialization control in Appendix F.

Reviewer_NXhROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Haidass, a 143M-parameter Chinese–English bilingual language model trained using a six-stage data curriculum. The authors investigate data quality filtering, language mixing, and training order, and demonstrate competitive performance in Chinese comprehension and bidirectional translation.

Critical strengths
  1. The systematic analysis of data quality, language ordering, and mixing ratios provides useful empirical insights into bilingual small language models.

  2. The experiments cover multiple model sizes and evaluation tasks, revealing both performance improvements and trade-offs.

  3. Haidass achieves competitive Chinese comprehension and translation performance with only 143M parameters. The detailed training recipe and evaluation protocols are also valuable.

Critical weaknesses

The optimality of the six-stage learning schedule is unclear. There is insufficient ablation analysis for this aspect. Would it be possible to compare the model against a fixed mixing baseline that uses fixed mixing proportions throughout training while matching the total number of tokens drawn from each data source?

Questions for authors

The optimality of the six-stage learning schedule is unclear. There is insufficient ablation analysis for this aspect. Would it be possible to compare the model against a fixed mixing baseline that uses fixed mixing proportions throughout training while matching the total number of tokens drawn from each data source?

2 comments

We sincerely thank the reviewer for the constructive suggestion to evaluate the six-stage curriculum against fixed-proportion mixing. At the full pretraining budget, the complete Haidass recipe improves five of eight tasks under both raw and length-normalized scoring. Together with the controlled filtering and language-interleaving studies, these results support the curriculum as an empirically motivated and effective training recipe. And for more detailed responses, please refer to the supplementary material Reply to Reviewer NXhR.pdf.

Reviewer comment: The optimality of the six-stage learning schedule is unclear. There is insufficient ablation analysis for this aspect. Would it be possible to compare the model against a fixed mixing baseline that uses fixed mixing proportions throughout training while matching the total number of tokens drawn from each data source?

Response: The fixed-mixture reference uses a single data-source distribution throughout training, with all 18 sources available from initialization. Ultra-FineWeb English, Ultra-FineWeb Chinese, and DCLM account for 39%, 27%, and 13%, respectively. The configured weights are applied to a blend of 4,096-token training samples. For a source with weight wiw_i, its planned token allocation is wi×391.1w_i\times391.1B. These allocations describe the intended training distribution, while realized source-wise consumption requires separate accounting.

In this reference, Cosmopedia accounts for a constant 5% of the mixture, and English L3 rewrites and question--answer data together account for 12%. The six-phase recipe introduces Cosmopedia in Phase 3 and assigns 40% and 60% to English L3 material in Phases 4 and 5, respectively. It also varies the main Chinese web-text weight across the first five phases, from 29% to 35% and subsequently to 26.3% and 18%, followed by a separate annealing mixture. The comparison therefore examines the complete staged recipe against an alternative with constant source weights and joint availability of its sources throughout training.

To assess the training design beyond the smaller-budget component studies, we compare the complete Haidass recipe with the fixed-mixture reference at the same reported budget of approximately 391.1B tokens. Haidass achieves higher raw accuracy on five of the eight evaluated tasks: ARC-Easy, ARC-Challenge, HellaSwag, PIQA, and ArithMark. The corresponding gains are 1.81, 0.94, 1.43, 1.47, and 1.90 percentage points. The same five tasks also improve under length-normalized scoring, showing that their observed advantage is retained under both scoring procedures.

Across all eight tasks, the unweighted mean raw accuracy increases from 38.62% for the fixed mixture to 39.42% for the curriculum-based recipe, a gain of 0.80 percentage points. Mean length-normalized accuracy increases from 41.01% to 41.87%, a gain of 0.86 points. Although OpenBookQA, WinoGrande, and AGIEval do not show uniform improvements, the aggregate results favor the proposed recipe under both scoring procedures. These findings extend the evidence for our training design to the full pretraining budget.

The controlled component studies provide complementary evidence for the underlying choices. Additional quality filtering improves the ten-task mean at all five evaluated model sizes while introducing task-dependent trade-offs (Section 4.1). Continuous bilingual interleaving better preserves balanced held-out language-modeling performance than the tested block schedules at matched per-language exposure (Section 4.2). These findings motivate combining quality-controlled data with sustained exposure to both languages, while the full-budget comparison shows that the complete recipe incorporating these principles achieves measurable aggregate gains over the tested fixed-mixture reference.

Taken together, the component studies inform individual data choices, and the full-budget results demonstrate improved aggregate performance for their integration into the proposed multi-stage recipe at the target training scale. This complementary evidence supports the curriculum as an empirically motivated and effective training recipe. We thank the reviewer for identifying the precise source-matched control needed to evaluate the independent contribution of the six-stage schedule.

Comments on the paper

1 comment

We sincerely thank the Area Chair and all three reviewers for their careful evaluation and constructive feedback. The reviewers raised important concerns regarding the effectiveness of data curation and language scheduling, the adequacy of controlled training comparisons, the generality of bilingual capabilities, and the fairness of translation baselines.

In response, we conducted additional controlled experiments on data filtering and language exposure order, introduced a full-budget fixed-mixture comparison at 391.1B tokens, expanded bilingual evaluations, and evaluated three similarly sized pretrained baselines (SmolLM2-135M, Cagliostro-v3-146M, and MobileLLM-140M) using the same three-stage translation adaptation strategy as Haidass-143M. The results show that quality filtering improves mean accuracy by 0.39–1.06 percentage points across five model sizes, continuous language interleaving reduces balanced held-out loss relative to blockwise scheduling, and the complete six-phase recipe improves mean raw accuracy by 0.80 points over the full-budget fixed-mixture reference; moreover, Haidass achieves 41.96% on C3, outperforming comparable sub-200M models, while the new translation comparison demonstrates the strongest English-to-Chinese performance (28.32 BLEU and 19.49 chrF++) and competitive Chinese-to-English performance (17.72 BLEU and 44.32 chrF++), only 0.29 BLEU below the best baseline.

Together with the substantial improvements over random initialization under matched translation fine-tuning, these findings provide stronger evidence that our data curation and language scheduling strategy supports effective compact bilingual pretraining, and that Haidass provides a competitive pretrained foundation for downstream bilingual translation rather than relying solely on task-specific adaptation. We sincerely appreciate the Area Chair and reviewers for their valuable suggestions, which have significantly strengthened the experimental evidence and clarified the contributions of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.