- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Haidass-143M, a Chinese–English bilingual language model trained on 391B tokens using a multi-stage natural-to-synthetic data curriculum. The authors investigate quality filtering, language mixing, and cross-lingual interference in small language models. Haidass achieves competitive performance among similarly sized models, while its translation variant approaches substantially larger models on FLORES benchmarks.
Critical strengths
- Relevant research problem: The paper investigates Chinese–English bilingual language modeling under tight parameter constraints (sub-200M), a relatively underexplored setting with practical relevance for resource-constrained deployment.
- Useful empirical analysis: Provides controlled experiments quantifying the effects of data filtering, language allocation, and scheduling across small model scales, highlighting task-dependent trade-offs and the benefits of continuous language interleaving.
Critical weaknesses
- Limited novelty and insufficient research contribution: The paper mainly combines established techniques without introducing a sufficiently novel methodology or research contribution for ICLR. Its empirical findings largely confirm known observations, with limited new scientific insights.
- Missing ablation of the main contribution: The proposed natural-to-synthetic curriculum is not compared against fixed-mixture or alternative scheduling strategies, making its actual contribution unclear.
- Limited model performance: The model scores near random chance on several bilingual benchmarks, providing little evidence of meaningful bilingual text-generation capabilities. Translation gains are largely limited to EN→ZH, while ZH→EN substantially underperforms NLLB. Moreover, Haidass uses tuned beam search while baselines use greedy decoding, raising fairness concerns.
Questions for authors
Given the limited capabilities of this 143M-parameter model, could the authors clarify the motivation for developing Haidass, its intended practical use cases, and its advantages in real-world deployment?
7 comments
Comment 1: Limited novelty and insufficient research contribution: The paper mainly combines established techniques without introducing a sufficiently novel methodology or research contribution for ICLR. Its empirical findings largely confirm known observations, with limited new scientific insights.
Response: We appreciate the reviewer's request for a more precise account of the research contribution. Our study examines how data selection and language exposure affect the capabilities retained by a compact Chinese--English model. We use an established decoder-only Transformer backbone so that the controlled studies can examine these data decisions under matched architectural and training settings within each model size.
Pretraining-data design is an established research problem with contributions grounded in controlled empirical analysis. DoReMi learns domain weights through a smaller proxy model, while RegMix predicts mixture performance from many small training runs . SmolLM2 combines dataset ablations with a documented small-model training recipe. Within this research direction, Haidass focuses on two questions: whether additional filtering benefits the same capabilities across compact model scales, and whether equal cumulative language exposure produces equal bilingual retention when its temporal order changes.
Section 4.1, Table 5, Figure 3, and Appendix C compare the full Ultra-FineWeb-en pool with its additional HQ subset, defined by fastText quality scores of at least 0.9. Both conditions therefore start from an already filtered web source. At each of five model sizes, from approximately 56M to 803M parameters, the comparison uses the same architecture, training hyperparameters, and approximately 20B training tokens. HQ improves the final ten-task mean at every evaluated size under both raw and length-normalized candidate scoring, by 0.39--1.06 and 0.75--1.23 percentage points, respectively. However, the gains are neither uniform across tasks nor monotonic in model size. At 469M, raw accuracy improves by 6.44 points on ARC-Easy and 3.67 on ARC-Challenge, while decreasing by 1.11 on HellaSwag and 0.54 on PIQA.
The resulting insight concerns what an aggregate improvement can conceal. A selection rule that improves the average can reduce particular capabilities, so its usefulness depends on the intended task profile. The cross-scale and task-level results provide a concrete basis for evaluating filtering through its capability trade-offs and for retaining complementary sources when designing a mixture. They also avoid assuming that a stronger filter or a larger model necessarily produces a larger benefit.
Prior multilingual research shows that language-modeling outcomes depend on data availability and model size. Our ordering experiment examines whether changing when each language is presented alters the final outcome when both languages receive the same cumulative token totals. Section 4.2, Figure 4(b), and Appendix D compare sequence-level IID interleaving, eight alternating monolingual blocks, and a single language switch in either direction. The 43M and 143M models use 100M and 331M training tokens, respectively, with a 50:50 allocation held fixed within each size. Eight-block alternation increases the equal-weight mean of the two languages' relative held-out losses by 4.57% and 4.62% compared with IID. These changes use byte-normalized packed-block loss and the corresponding same-size IID reference, as defined in Appendix D, Eq. (3).
The single-switch results reveal a larger and asymmetric retention cost. At 143M, English-then-Chinese training increases English loss by 38.22% relative to IID, whereas the reverse order increases Chinese loss by 74.02%. In both directions, the language encountered first suffers the larger deterioration despite receiving the same cumulative exposure as under IID. This distinguishes the ordering effect from the separate language-ratio experiment, in which changing the ratio necessarily changes the training tokens allocated to each language.
These experiments add quantitative guidance for the studied compact bilingual setting: task-level performance is needed to interpret filtering gains, and final language-token totals are insufficient to characterize retention. The evidence supports assessing data selection against the intended capabilities and maintaining interleaved exposure when balanced language retention is required. The released model and documented recipe provide a concrete setting for applying and further examining these findings, with task-specific adaptation demonstrating their practical relevance.
Comment 2: Missing ablation of the main contribution: The proposed natural-to-synthetic curriculum is not compared against fixed-mixture or alternative scheduling strategies, making its actual contribution unclear.
Response: Thank you for requesting stronger experimental support for the training design. We report two complementary comparisons: a fixed-mixture reference at the released model's total token budget and a comparison of the initial data sources under matched training settings. The recipe progressively introduces rewritten and question--answer text while retaining web text; its motivation is to combine an informative initial source with additional ways of presenting knowledge within the same next-token prediction objective.
The fixed-mixture reference was trained for 391.1B processed tokens. At this full budget, the Haidass recipe achieves higher raw accuracy on five of the eight evaluated English tasks: ARC-Easy, ARC-Challenge, HellaSwag, PIQA, and ArithMark. The respective improvements are 1.81, 0.94, 1.43, 1.47, and 1.90 percentage points. The same five tasks also improve under length-normalized scoring, so the observed advantage is retained under both evaluation procedures.
Across all eight tasks, the unweighted mean raw accuracy increases from 38.62% for the fixed mixture to 39.42% for our curriculum-based recipe, an improvement of 0.80 percentage points. Mean length-normalized accuracy increases from 41.01% to 41.87%, an improvement of 0.86 points. Although OpenBookQA, WinoGrande, and AGIEval do not show uniform improvements, the overall results favor the proposed recipe under both scoring procedures.
These findings extend the evidence for our training design from controlled component studies to the full pretraining budget. The smaller-budget experiments show that quality filtering improves aggregate performance while introducing task-dependent trade-offs, and that continuous language interleaving better preserves balanced bilingual performance under matched per-language exposure. The full-budget comparison further shows that the complete recipe incorporating these principles achieves measurable aggregate gains over the tested fixed-mixture reference. Controlled experiments thus motivate the individual data choices, while the full-budget evaluation demonstrates improved aggregate performance when they are integrated into the proposed multi-stage recipe at the target training scale.
We also train three 143M models from random initialization on the original Ultra-en web recipe, L3 multi-style rewrites, or L3 question--answer data for approximately 20B tokens each. They share the architecture, tokenizer, sequence length, global training batch size, learning-rate configuration, and training seed. The original web recipe achieves the highest four-task mean under both scoring procedures: 44.26% raw accuracy and 43.95% normalized accuracy, compared with 41.42%/41.44% for L3 multi-style and 42.24%/42.07% for L3 QA. Its normalized mean also exceeds both alternatives at the shared checkpoints of approximately 5.24B, 10.49B, and 15.73B tokens. The task-level results retain complementary behavior: both L3 recipes improve normalized ARC-Challenge, and L3 QA slightly improves raw PIQA.
This experiment provides direct support for the initial web-source choice among the tested recipes. It compares sources from initialization, rather than exchanging their order at identical cumulative source exposure, and therefore establishes early-learning behavior without determining the optimal transition time for later data. The subsequent use of rewritten text has a separate rationale: WRAP shows that combining web documents with synthetic rephrasings can improve pretraining efficiency and downstream performance in its evaluated settings . In Haidass, multi-style and question--answer sources provide alternative presentations of information while preserving the next-token prediction objective described in Section-2.2.
Smaller controlled studies examine individual choices at the evaluated scale. Data Mixing Laws shows that mixture rankings can change with model size and training duration . We therefore use the component experiments to characterize filtering, exposure order, and initial-source effects, and assess the complete recipe separately at 391.1B reported tokens. A schedule-only comparison would additionally hold the tokenizer, source data, and cumulative source exposure fixed. Together, the results support an initial web foundation, continued bilingual exposure, and complementary text forms: the original web recipe gives the best average early performance among the tested sources, interleaving favors balanced retention at matched exposure, and the complete recipe improves five evaluations at the full reported budget.
We sincerely thank the reviewer for the careful assessment of the scientific contribution, curriculum evidence, and practical value of Haidass. These comments have helped us clarify how the controlled experiments inform compact Chinese--English training. In the responses below, we relate the contribution to prior research on pretraining data, present the full-budget and initial-source comparisons, and explain what the bilingual and translation evaluations establish.
The experiments support several concrete findings. Additional quality filtering improves mean accuracy across five model sizes while producing task-dependent losses; interleaving preserves balanced language performance better than the tested block schedules at matched per-language exposure; and the original web recipe gives the highest four-task mean among the tested initial sources. At the reported 391.1B-token budget, the complete Haidass recipe improves accuracy on five of eight tasks relative to the fixed-mixture reference. The capability evaluations further show a Chinese reading-comprehension and text-likelihood advantage among the similarly sized references, together with competitive task-specific translation after post-training. We explain the scope and practical implications of each result below and for more detailed responses, please refer to the supplementary material Reply to Reviewer ZkE6.pdf