acceptodds
Under review as a conference paper at ICLR 2027

NuosuBench: Revealing the Limits of Multilingual Large Language Models in Standardized Yi

Abstract

Multilingual large language models (LLMs) exhibit strong cross-lingual transfer across many languages, but it remains unclear whether this capability extends to languages written in severely underrepresented scripts. We investigate this question through Standardized Yi (Nuosu), a language spoken by millions of people in southwestern China but largely absent from existing LLM training and evaluation ecosystems. We introduce Nuosu-Datasets and Nuosu-Benchmark, the first large-scale resource suite for training and systematically evaluating Standardized Yi language models. Nuosu-Datasets contains 78.6M tokens for continual pre-training and 166M tokens comprising 1.1M instruction instances for supervised fine-tuning and cross-lingual alignment. The data are collected from authoritative web and expert-provided sources through OCR-based digitization, multi-stage filtering, and expert validation. Nuosu-Benchmark contains 110513 instances across five sub-benchmarks that evaluate lexical knowledge, sentence-level understanding, culturally grounded knowledge, administrative language, and educational comprehension using 12 complementary metrics. Evaluating 14 representative LLMs reveals a substantial gap between general multilingual capability and competence in Standardized Yi. All evaluated general-purpose models achieve aggregate scores below 40%, indicating that model scale and broad multilingual training alone do not ensure effective transfer to an underrepresented script. In contrast, BIMO-8B, an 8B model adapted using Nuosu-Datasets, achieves 50.1%, outperforming the strongest evaluated general-purpose model by 10.2 percentage points. Further analyses show that language-specific training produces improvements across multiple levels of linguistic competence, while culturally grounded and knowledge-intensive tasks remain particularly challenging. Our resources and findings establish a reproducible foundation for studying cross-lingual transfer, data-efficient adaptation, and evaluation in extremely low-resource language modeling.

Primary area:
datasets and benchmarks
Keywords:
Low-resource language modeling, Multilingual large language models, Standardized Yi, Nuosu, Dataset construction, Benchmark evaluation, Cross-lingual transfer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9jJaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates cross-lingual transfer through Standardized Yi (Nuosu), a language spoken by millions of people in southwestern China but largely absent from existing LLM training and evaluation ecosystems. The data are collected from authoritative web and expert-provided sources through OCR-based digitization, multi-stage filtering, and expert validation. The authors build a benchmark, Nuosu-Benchmark, containing 110,513 instances across five sub-benchmarks that evaluate lexical knowledge, sentence-level understanding, culturally grounded knowledge, administrative language, and educational comprehension using 12 complementary metrics. This paper evaluates 14 representative LLMs and reveals a substantial gap between general multilingual capability and competence in Standardized Yi.

Critical strengths
  1. The paper investigated an underrepresented language and released a dataset, a benchmark, and adapted models.
  2. The paper used a series of data engineering tools and models to curate the dataset, with details reported.
  3. The paper benchmarked a list of representative models and reported a few metrics across tasks.
Critical weaknesses
  1. The statement "largely absent from existing LLM training and evaluation ecosystems" is a good selling point. But the corpus has a category called "Official documents". There might be a chance that those official documents have already been explored by existing studies and model training (especially multilingual LLMs adapted for low-resource languages).
  2. Following the "official documents", both CPT corpus and the benchmark contain the Yi editions of People's Daily. So there is possible train-test contamination between training and benchmarking. This should be addressed.
  3. There are no ablations in model adaptation; some missing controls are, e.g., CPT-only versus SFT-only versus CPT+SFT.
  4. The expert validation is not detailed, missing annotator profiles and qualifications, and validation protocols. Translated data and synthetic or extracted data should be better audited at each stage.
Questions for authors
  1. Is each pivot-augmented Yi–Zh–En sample a separate unit in the split, or a field added to an existing Yi–Zh unit?
  2. What did the "expert validation" consist of? Can you provide details?
  3. Why does BIMO-8B score below Qwen3-8B on NuosuSen?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 9jJa

Thank you for your valuable comments.

Q1. Official documents and train-test contamination

We clarify the roles of monolingual and bilingual sources, and the contribution of organizing publicly available materials into training and evaluation resources.

CPT sources and their role. Our CPT corpus consists primarily of monolingual Standardized Yi materials, including Yi-only books, online tutorials written entirely in Yi, reports, prose, and other web texts. These records lack corresponding Chinese translations in our collection and are therefore used for next-token pre-training to develop Yi representations. The bilingual collections provide the aligned source material for cross-lingual SFT and benchmark construction. NuosuGov draws primarily on official bilingual materials provided by Southwest Minzu University for public use, together with bilingual material from People’s Daily. Bilingual records assigned to SFT and evaluation follow disjoint underlying source-unit allocations, and all derivatives inherit their source unit’s allocation.

Public documents and prior model exposure. Publicly accessible Yi documents may have been included in the pre-training corpora of models such as Qwen. Our evaluation nevertheless reveals extremely weak practical Yi competence in Qwen3-8B: its Nuosu-Benchmark score is 29.58, below the 31.95 average across the 14 evaluated models. The gap between potential exposure and measured competence motivates systematic language-specific resources. Our contribution is to organize scattered materials into standardized CPT and SFT corpora, construct tasks across five linguistic and usage levels, and establish a unified benchmark with reproducible evaluation. These resources make the collected materials usable for language adaptation and systematic capability assessment.

Q2. Partitioned units of pivot-augmented samples

An augmented Yi–Zh–En example is stored and counted as an additional record, with the English text derived from the Chinese side of its original Yi–Zh pair. The underlying Yi–Zh source unit is the unit of assignment. The original pair and its augmented record share the same split, as do any instruction examples subsequently generated from that source. We assign the records to the training and evaluation pools before applying TIEA. This means an original Yi–Zh pair cannot be assigned to SFT while its Yi–Zh–En version is assigned to evaluation.

Q3. Why does BIMO-8B score below Qwen3-8B on NuosuSen

BIMO-8B scored 29.06 on NuosuSen, 3.63 points below Qwen3-8B. A plausible explanation is a mismatch in register. NuosuSen includes sentence translation and colloquial expressions, while everyday expressions account for only 6.47% of the parallel corpus. Classical verse and government reports account for 37.58% and 23.72%, respectively. The monolingual Yi corpus used for CPT is also dominated by news, which accounts for 79.54% of its tokens. These distributions show that the training data provide much more exposure to formal and literary registers than to everyday expressions.

In the follow-up study, we grouped the Yi CPT data by difficulty, introduced curriculum sampling, retained the original vocabulary, and included general-purpose data at 20% of the total tokens in each training stage. The resulting Nuosu-8B scored 54.88 on NuosuSen, above both BIMO-8B and Qwen3-8B in the reported evaluations. The follow-up shows that sentence-level performance can improve substantially under a revised training recipe. The corpus distribution makes limited coverage of everyday expressions a plausible contributor to the original regression.

Q4. What did the "expert validation" consist of? Can you provide details?

At an earlier stage of processing, two Yi studies experts proofread the OCR outputs against the source scans. For each sub-benchmark, two Yi studies experts then jointly reviewed the items for factual and linguistic consistency and assessed whether answering each item required the skill it was designed to assess. They discussed uncertain cases until they reached consensus and recorded one joint judgment per item.

Two NuosuEdu items illustrate how the experts applied these criteria. For a translation item drawn from 《中国彝文典籍译丛》, they confirmed that 祭后酿恶果, accurately conveys the meaning of ꄰꉩꀋꏜꉩ,. For a cloze item with the Chinese gloss 众神纷纷归将去。, they checked that the reference ꃅꇑꋍꀕꃅ。 preserves every visible character in _ꇑ_ꀕꃅ_ and matches the gloss. The translation example illustrates how the experts checked semantic fidelity, while the cloze example illustrates how they checked both meaning and adherence to the prompt’s constraints.

AuthorsOpenReview ↗

Response to Reviewer 9jJa

Q5. Training ablations

The primary contribution of this paper is the construction of Nuosu-Datasets and Nuosu-Benchmark. BIMO-8B demonstrates the value of these resources for language adaptation. We report the training-stage comparisons below, followed by the results of Nuosu-8B, our latest checkpoint trained with a revised recipe.

Contribution of CPT and SFT. We now report the base model, CPT-only, SFT-only, and CPT+SFT results. CPT uses a next-token prediction objective on monolingual Yi data mixed with general-purpose text, while SFT introduces cross-lingual instruction supervision. The monolingual Yi materials include Yi-only books, online tutorials written entirely in Yi, reports, and prose. These records lack corresponding Chinese translations in our collection and are therefore used for CPT.

Training configurationNuosu-Benchmark score
Qwen3-8B, before adaptation29.58
CPT-only34.83
SFT-only, LoRA33.25
CPT+SFT, LoRA45.55
SFT-only, full-parameter fine-tuning37.27
CPT+SFT, full-parameter fine-tuning50.11

The CPT-only results below follow the metric order in Table 5. Their unweighted mean is 34.8275, reported as 34.83 points.

ModelBLEU-1BLEU-2BLEU-3BLEU-4CIDErGLEUMETEORNISTROUGEchrFchrF++1/(1+TER)
CPT-only21.1315.2213.9911.64103.0215.2122.67112.8125.3315.6613.3047.95

CPT alone improves the composite score over Qwen3-8B by 5.25 points. Adding CPT before SFT improves the score by 12.30 points with LoRA and 12.84 points with full-parameter fine-tuning. The full-parameter CPT+SFT model also exceeds CPT-only by 15.28 points, showing the complementary contributions of target-language pre-training and cross-lingual instruction tuning.

For Nuosu-8B, we grouped the Yi CPT data by difficulty and designed a curriculum sampling function that gradually introduced more difficult material. Both vocabulary configurations used the same curriculum and included general-purpose data accounting for 20% of the total training tokens in each stage, calculated separately for CPT and SFT.

Follow-up configurationNuosu-Benchmark scoreGeneral-capability average
Curriculum sampling with an expanded vocabulary57.6660.22
Curriculum sampling with the original vocabulary (Nuosu-8B)59.4868.90

Under this curriculum, retaining the original vocabulary improved the Nuosu-Benchmark score by 1.82 points and the general-capability average by 8.68 points compared with expanding it. Nuosu-8B also scored 54.88 on NuosuSen, exceeding the reported Qwen3-8B score of 32.69. The revised recipe thus reverses the sentence-level regression observed in the submitted run.

General-capability retention. BIMO-8B declined on all five general benchmarks, a pattern consistent with forgetting after adaptation. Nuosu-8B improves on all five in the reported evaluations.

BenchmarkQwen3-8BBIMO-8BNuosu-8B
MMLU87.5074.4081.30
GPQA62.0038.2055.70
LiveBench67.1048.1060.30
MATH-50097.4051.2080.60
AIME’2567.3031.3066.60
Average76.2648.6468.90

The largest gains over BIMO-8B are on AIME’25 (+35.30 points) and MATH-500 (+29.40 points). Nuosu-8B comes within 0.70 points of Qwen3-8B on AIME’25, while MATH-500 remains the largest gap at 16.80 points. Across the five benchmarks, the average gap to Qwen3-8B narrows from 27.62 to 7.36 points. The follow-up recipe therefore mitigates the loss of general capability while improving both sentence-level Yi performance and the overall Nuosu-Benchmark score.

AuthorsOpenReview ↗

Response to Reviewer 9jJa

Q6. Insufficient benchmark validation

We will describe the checks applied at each data-processing stage and the final human review in the main paper.

StageChecking procedure
OCR extractionTwo Yi studies experts proofread the recognized text against the scanned source documents.
Pivot translationEnglish was generated automatically from the Chinese side of the original Yi–Chinese record. The augmented records underwent automatic checks within the processing pipeline and retained their original source-unit assignments.
TIEA generationTasks were produced through programmed templates and transformations and checked automatically. The checks covered available answer labels, preservation of unmasked characters, and preservation of character counts in reordering tasks.
Final benchmark reviewTwo doctoral students from SWUFE first checked item formatting and content against the cross-lingual source data. Two Yi studies experts from SWUN then jointly reviewed the items for factual and linguistic consistency and the skill elicited by each task.The expert review used a joint consensus mechanism: uncertain cases were discussed until the experts agreed on one judgment per item.

Taking NuosuEdu as an example:

Quantitative evidence.

TaskItems (% of NuosuEdu)Findings
Translation16,886 (22.82%)5,709 Chinese-to-Yi, 5,926 Yi-to-Chinese, 2,602 English-to-Yi, and 2,649 Yi-to-English items.
Multiple choice15,970 (21.58%)All 15,970 reference labels point to an available option. In 15,969 items, all option texts are distinct.
Cloze4,925 (6.66%)All 4,925 prompts have the same character length as their references and preserve every unmasked character. The prompts mask 33.21% of characters on average.
Translation judgment27,384 (37.01%)13,776 positive labels (50.31%) and 13,608 negative labels (49.69%).
Error correction5,596 (7.56%)The reference changes the input text in 3,484 items. Among the 2,112 items with unchanged text, 2,111 explicitly indicate that no correction is needed.
Character reordering3,231 (4.37%)All 3,231 prompts preserve the reference characters and their counts. In 3,226 items, the character order differs from the reference.

Across the 15,970 multiple-choice items, uniform guessing gives an expected accuracy of 15.67%, computed using the actual number of options in each item. Always selecting the most frequent answer position, B, gives 15.95%. This comparison checks one simple answer-position heuristic. A separate check found 73,987 distinct full records after normalizing Unicode and whitespace, leaving five redundant records in the released NuosuEdu set.

validation examples. Translation and multiple-choice items ask models to produce or recognize cross-lingual equivalents of words and fixed expressions. Judgment items ask whether a proposed translation preserves the source meaning. Cloze, correction, and character-reordering items ask models to recover a target expression under explicit constraints. For example, the experts confirmed 祭后酿恶果, as the Chinese translation of ꄰꉩꀋꏜꉩ,, an item drawn from 《中国彝文典籍译丛》. A reviewed cloze item asks the model to complete _ꇑ_ꀕꃅ_ using a Chinese gloss. Its reference, ꃅꇑꋍꀕꃅ。, preserves the visible characters and supplies the missing ones. The appendix will provide the full quantitative evidence for Nuosu-Benchmark and examples from the expert validation.

Reviewer_gNSROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Nuosu-Datasets (244.6M tokens) and Nuosu-Benchmark (110K instances across five tasks) for Standardized Yi, an underrepresented language. Evaluating 14 LLMs, the authors show that their adapted Qwen3-8B model (BIMO-8B) substantially outperforms general-purpose models, highlighting the importance of language-specific adaptation.

Critical strengths
  1. Underrepresented language coverage: Addresses a severely underrepresented language (Standardized Yi) with significant cultural and linguistic relevance, contributing to its inclusion in multilingual LLM research.
  2. Training datasets and benchmark: Introduces CPT and SFT training datasets, alongside a comprehensive, multi-level benchmark covering lexical, syntactic, cultural, and domain-specific language understanding.
Critical weaknesses
  1. Catastrophic forgetting: BIMO-8B substantially degrades on general benchmarks compared to Qwen3-8B (76.26 → 48.64 average), despite SFT including Chinese/English alignment data. This degradation raises concerns about the method's effectiveness and practical utility, yet the paper does not address this issue.
  2. Shared training-evaluation pipeline and potential evaluation bias: The same TIEA pipeline is used to generate both the SFT data and NuosuEdu (67% of evaluation instances). Although the authors employ disjoint source-level splits to prevent direct overlap, shared task formats and templates may favor models trained on the generated data. The paper does not sufficiently investigate this risk or provide a comprehensive decontamination analysis beyond source-level separation.
  3. Insufficient benchmark validation: Despite claiming expert validation, the paper provides no systematic human quality assessment or reported verification of the synthetically generated evaluation instances, raising concerns about benchmark reliability.
  4. Limited corpus size: The datasets are relatively small, particularly the CPT corpus, which contains only 78.6M tokens and is used for five training epochs. This raises concerns about data diversity, overfitting, and generalization.
Questions for authors
  • Figure 1 is difficult to read. Please enlarge or restructure it to improve clarity.
  • Clarify how the evaluation metrics are aggregated into sub-benchmark scores and the overall score.
2 comments
AuthorsOpenReview ↗

Response to Reviewer gNSR

Thank you for your valuable comments.

Q1. Catastrophic forgetting

We agree that the decline in general capability limits the practical use of the submitted BIMO-8B. Its average falls from 76.26 to 48.64 relative to Qwen3-8B, with declines on all five general benchmarks. This pattern is consistent with forgetting after adaptation. The Chinese and English alignment examples in SFT primarily teach cross-lingual correspondence with Yi. They provide limited practice on the general knowledge, mathematical reasoning, and other skills assessed by these benchmarks.

In a follow-up study using the same Nuosu resources, we trained Nuosu-8B with a revised recipe. We grouped the Yi CPT data by difficulty, used curriculum sampling, retained the original vocabulary, and mixed in general-purpose data at both training stages. General-purpose data accounted for 20% of the total tokens in each stage, calculated separately for CPT and SFT.

General benchmarkQwen3-8BBIMO-8BNuosu-8B
MMLU87.5074.4081.30
GPQA62.0038.2055.70
LiveBench67.1048.1060.30
MATH-50097.4051.2080.60
AIME’2567.3031.3066.60
Average76.2648.6468.90

Nuosu-8B scores higher than BIMO-8B on all five general benchmarks, raising the average by 20.26 points and narrowing the gap to Qwen3-8B from 27.62 to 7.36 points. Retention varies by task. Nuosu-8B comes within 0.70 points of Qwen3-8B on AIME’25, while MATH-500 remains 16.80 points below the base model. At the same time, Nuosu-8B reaches 59.48 on Nuosu-Benchmark, including 54.88 on NuosuSen. The follow-up results show that the large general-capability loss observed in the submitted run can be reduced while improving performance on Yi tasks.

Q2. Shared training-evaluation pipeline and potential evaluation bias

Training and evaluation use disjoint source content. We split the source material before applying TIEA. Each generated item inherits the split of its source unit, so alternative prompts and other derivatives of the same source stay on the same side of the split. TIEA therefore applies some of the same task formats to distinct source content in the SFT data and NuosuEdu. We will clarify this distinction in the main text.

BIMO-8B’s lead persists on tasks constructed without TIEA. Appendix J.1, Table 8 directly examines whether the lead depends on including NuosuEdu. We excluded NuosuEdu entirely and compared all 14 models on NuosuLex, NuosuSen, NuosuEpic, and NuosuGov under the same 10-shot protocol. These four subsets were constructed without TIEA and retain their original references. With the same 12-metric scoring rule applied to every model, BIMO-8B ranks first on the four-subset mean at 49.99 points, ahead of Gemini 2.5-Pro (41.66) and Claude 4 Opus (39.19). Its margin over the strongest general-purpose model is 8.33 points.

This comparison shows that BIMO-8B’s leading rank does not depend on the inclusion of TIEA-generated evaluation items. Any benefit from template familiarity within NuosuEdu is therefore insufficient to explain its lead on the four subsets constructed without TIEA.

Q3. Limited corpus size

The public low-resource datasets we examined contained very little Standardized Yi text. We therefore collected monolingual material from official publications and authoritative Yi textual collections. The resulting 78.6M-token CPT corpus was the largest we could assemble from sources that we could verify were suitable for model training. It includes news (79.54%), literature (11.84%), classical texts (7.38%), and government reports (1.24%). These sources cover several genres of Yi.

We agree that five passes over a fixed corpus raise a concern about repeated exposure. In our follow-up study, we trained Nuosu-8B for three CPT epochs, grouped the Yi texts by difficulty, and used curriculum sampling to introduce more difficult material as training progressed. We retained the original vocabulary and included general-purpose data at 20% of the total token count in each stage, calculated separately for CPT and SFT.

With this recipe, Nuosu-8B reached 59.48 on Nuosu-Benchmark, including 54.88 on NuosuSen. Its general-capability average rose from 48.64 for BIMO-8B to 68.90, narrowing the gap to Qwen3-8B from 27.62 to 7.36 points. The follow-up thus shows that three CPT epochs can support stronger Yi performance and substantially better retention of general capabilities when combined with the revised sampling and data mixture.

We expect that adding new Yi text, especially everyday and conversational material, would broaden the corpus and give the model more exposure to the kinds of sentences evaluated in NuosuSen.

Q4. Figure 1 Readability

Thanks for the suggestion, we have revised Figure 1 in the main paper to improve readability.

AuthorsOpenReview ↗

Response to Reviewer gNSR

Q5. Insufficient benchmark validation

We will describe the checks applied at each data-processing stage and the final human review in the main paper.

StageChecking procedure
OCR extractionTwo Yi studies experts proofread the recognized text against the scanned source documents.
Pivot translationEnglish was generated automatically from the Chinese side of the original Yi–Chinese record. The augmented records underwent automatic checks within the processing pipeline and retained their original source-unit assignments.
TIEA generationTasks were produced through programmed templates and transformations and checked automatically. The checks covered available answer labels, preservation of unmasked characters, and preservation of character counts in reordering tasks.
Final benchmark reviewTwo doctoral students from SWUFE first checked item formatting and content against the cross-lingual source data. Two Yi studies experts from SWUN then jointly reviewed the items for factual and linguistic consistency and the skill elicited by each task.The expert review used a joint consensus mechanism: uncertain cases were discussed until the experts agreed on one judgment per item.

Taking NuosuEdu as an example:

Quantitative evidence.

TaskItems (% of NuosuEdu)Findings
Translation16,886 (22.82%)5,709 Chinese-to-Yi, 5,926 Yi-to-Chinese, 2,602 English-to-Yi, and 2,649 Yi-to-English items.
Multiple choice15,970 (21.58%)All 15,970 reference labels point to an available option. In 15,969 items, all option texts are distinct.
Cloze4,925 (6.66%)All 4,925 prompts have the same character length as their references and preserve every unmasked character. The prompts mask 33.21% of characters on average.
Translation judgment27,384 (37.01%)13,776 positive labels (50.31%) and 13,608 negative labels (49.69%).
Error correction5,596 (7.56%)The reference changes the input text in 3,484 items. Among the 2,112 items with unchanged text, 2,111 explicitly indicate that no correction is needed.
Character reordering3,231 (4.37%)All 3,231 prompts preserve the reference characters and their counts. In 3,226 items, the character order differs from the reference.

Across the 15,970 multiple-choice items, uniform guessing gives an expected accuracy of 15.67%, computed using the actual number of options in each item. Always selecting the most frequent answer position, B, gives 15.95%. This comparison checks one simple answer-position heuristic. A separate check found 73,987 distinct full records after normalizing Unicode and whitespace, leaving five redundant records in the released NuosuEdu set.

validation examples. Translation and multiple-choice items ask models to produce or recognize cross-lingual equivalents of words and fixed expressions. Judgment items ask whether a proposed translation preserves the source meaning. Cloze, correction, and character-reordering items ask models to recover a target expression under explicit constraints. For example, the experts confirmed 祭后酿恶果, as the Chinese translation of ꄰꉩꀋꏜꉩ,, an item drawn from 《中国彝文典籍译丛》. A reviewed cloze item asks the model to complete _ꇑ_ꀕꃅ_ using a Chinese gloss. Its reference, ꃅꇑꋍꀕꃅ。, preserves the visible characters and supplies the missing ones. The appendix will provide the full quantitative evidence for Nuosu-Benchmark and examples from the expert validation.

Q6. Metric Aggregation

For each model, we take the mean of the 12 metric values within each sub-benchmark and then take the mean of five sub-benchmarks as the overall score.

\qquad S=\frac{1}{5}\sum_{b=1}^{5}S_b,$$ where $m_{b,j}$ is the value of metric $j$ on sub-benchmark $b$, $S_b$ is the sub-benchmark score, and $S$ is the overall score. We agree that “percentage points” was inappropriate and will report differences in **points**. Metrics on larger scales can still contribute more to score differences. To assess this sensitivity, we removed NIST and CIDEr and the results are as follows: | Model |12 metrics | Rank | Without NIST and CIDEr | Rank | |---|---:|---:|---:|---:| | **BIMO-8B (Ours)** | **50.11** | **1** | **29.09** | **1** | | Claude 4 Opus | 39.87 | 2 | 20.67 | 3 | | Gemini 2.5-Pro | 39.40 | 3 | 21.81 | 2 | | LLaMA4-Maverick | 38.79 | 4 | 19.82 | 4 | | GPT-5 | 37.45 | 5 | 18.91 | 6 | | Qwen3-235B | 37.44 | 6 | 19.59 | 5 | | DouBao-Seed-1.6 | 36.27 | 7 | 18.05 | 7 | | DeepSeek-v3.1 | 36.19 | 8 | 17.63 | 10 | | Qwen3-8B | 29.58 | 9 | 17.98 | 8 | | Ministral-8B | 26.80 | 10 | 17.84 | 9 | | LLaMA3.1-8B | 25.12 | 11 | 17.32 | 11 | | GLM-4.5 | 22.78 | 12 | 14.29 | 12 | | Kimi-K2-0905 | 15.65 | 13 | 11.24 | 13 | | Gemma3n-E4B | 11.91 | 14 | 10.52 | 14 | **BIMO-8B remains first among all 14 models**, while several other ranks change.We will report the scoring rule and sensitivity finding in the main paper.
Reviewer_4JmnOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Nuosu-Datasets and Nuosu-Benchmark, a large-scale resource suite for Standardized Yi, an underrepresented language and writing system. The authors construct corpora for continual pre-training and supervised fine-tuning, alongside a benchmark comprising 110,513 instances across five linguistic levels. They evaluate 14 language models and develop BIMO-8B through language-specific adaptation, reporting improvements over general-purpose LLMs.

Critical strengths
  1. The paper addresses an important and underserved language. The datasets are a valuable resource for the research community and could support further work on Nuosu and other low-resource scripts.
  2. The data engineering effort is substantial: web crawling, a fine-tuned Yi OCR model for scanned documents, and multi-stage cleaning of parallel data. Code and data are released.
  3. The benchmark includes five task categories and 12 evaluation metrics, covering lexical, character-level, and edit-distance measurements. It evaluates 14 models and compares general-purpose and language-adapted approaches. Separating SFT and evaluation data and excluding the synthetic NuosuEdu subset from the analysis strengthens the evaluation methodology.
Critical weaknesses
  1. The main paper does not adequately explain how the 12 heterogeneous evaluation metrics relate to the sub-benchmark scores and overall performance. While the appendix indicates that the overall score averages five sub-benchmark scores, the construction and interpretation of these scores remain unclear, especially given the differing scales of the metrics. This makes the claim that BIMO-8B improves performance by 10.24 percentage points difficult to interpret. As this is a key finding, the main paper should include more methodological detail rather than relying primarily on supplementary material.
  2. The benchmark validation requires stronger evidence. NuosuEdu contains 73,992 instances generated from 8,110 vocabulary-level pairs using templates. While this adds task diversity, it is unclear whether these synthetic tasks adequately measure the claimed linguistic understanding and reasoning capabilities. The paper also provides limited quantitative evidence of annotation reliability and does not sufficiently document independent human validation of the generated evaluation instances.
Questions for authors
  1. Could the authors explicitly explain how the 12 evaluation metrics are combined into each sub-benchmark score, justify their aggregation across different scales, and clarify why the reported overall improvements are expressed as percentage points?
  2. How was the validity of the synthetic NuosuEdu evaluation established? Could the authors provide a manually validated sample or additional analysis demonstrating that its tasks measure the intended linguistic and reasoning capabilities?
  3. Could the authors provide CPT-only, SFT-only, and CPT+SFT ablations to clarify the contribution of each training stage? Additionally, could they investigate why BIMO-8B performs worse than its base model, Qwen3-8B, on NuosuSen (29.06 vs. 32.69) and general-capability benchmarks (48.64 vs. 76.26), and whether these declines indicate negative transfer or catastrophic forgetting?
  4. Could the authors clarify the expert-validation procedure, including the number and qualifications of annotators, validation coverage, and inter-annotator agreement where applicable?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 4Jmn

Thank you for your valuable comments.

Q1 . Metric Aggregation and Interpretation

For each model, we take the mean of the 12 metric values within each sub-benchmark and then take the mean of five sub-benchmarks as the overall score.

\qquad S=\frac{1}{5}\sum_{b=1}^{5}S_b,$$ where $m_{b,j}$ is the value of metric $j$ on sub-benchmark $b$, $S_b$ is the sub-benchmark score, and $S$ is the overall score. We agree that “percentage points” was inappropriate and will report differences in **points**. Metrics on larger scales can still contribute more to score differences. To assess this sensitivity, we removed NIST and CIDEr and the results are as follows: | Model |12 metrics | Rank | Without NIST and CIDEr | Rank | |---|---:|---:|---:|---:| | **BIMO-8B (Ours)** | **50.11** | **1** | **29.09** | **1** | | Claude 4 Opus | 39.87 | 2 | 20.67 | 3 | | Gemini 2.5-Pro | 39.40 | 3 | 21.81 | 2 | | LLaMA4-Maverick | 38.79 | 4 | 19.82 | 4 | | GPT-5 | 37.45 | 5 | 18.91 | 6 | | Qwen3-235B | 37.44 | 6 | 19.59 | 5 | | DouBao-Seed-1.6 | 36.27 | 7 | 18.05 | 7 | | DeepSeek-v3.1 | 36.19 | 8 | 17.63 | 10 | | Qwen3-8B | 29.58 | 9 | 17.98 | 8 | | Ministral-8B | 26.80 | 10 | 17.84 | 9 | | LLaMA3.1-8B | 25.12 | 11 | 17.32 | 11 | | GLM-4.5 | 22.78 | 12 | 14.29 | 12 | | Kimi-K2-0905 | 15.65 | 13 | 11.24 | 13 | | Gemma3n-E4B | 11.91 | 14 | 10.52 | 14 | **BIMO-8B remains first among all 14 models**, while several other ranks change.We will report the scoring rule and sensitivity finding in the main paper. # Q2 and Q4. NuosuEdu Validity and Human Validation **Human validation.** For each sub-benchmark, two doctoral students from SWUFE checked item formatting and content against the source data and screened for duplicates. Two Yi studies experts from SWUN then checked factual and linguistic consistency and assessed whether the items elicited the intended skills. The experts did not provide independent ratings, so we could not estimate inter-annotator agreement. For NuosuEdu, the experts jointly reviewed all 73,992 items and resolved uncertain cases through discussion, recording one consensus judgment per item. Translation and multiple-choice items ask models to produce or recognize cross-lingual equivalents of words and fixed expressions. Judgment items ask whether a proposed translation preserves the source meaning. Cloze, correction, and character-reordering items ask models to recover a target expression under explicit constraints. The experts confirmed `祭后酿恶果,` as the Chinese translation of `ꄰꉩꀋꏜꉩ,`, an item drawn from 《中国彝文典籍译丛》. A reviewed cloze item asks the model to complete `_ꇑ_ꀕꃅ_` using a Chinese gloss. Its reference, `ꃅꇑꋍꀕꃅ。`, preserves the visible characters and supplies the missing ones. **Checks at different processing stages.** Two Yi studies experts proofread OCR-extracted Yi text against the scanned source documents. Pivot augmentation and TIEA generation were checked automatically during processing. Pivot augmentation used machine translation to produce English from the Chinese side of an existing Yi–Chinese pair, retaining the original pair’s source allocation. TIEA generated tasks through programmed templates and transformations, with automatic checks on the resulting item structure. The final benchmark items were then reviewed using the human consensus procedure described above. **Quantitative evidence for NuosuEdu Validity.** | Task | Items (% of NuosuEdu) | Findings | |---|---:|---| | Translation | 16,886 (22.82%) | 5,709 Chinese-to-Yi, 5,926 Yi-to-Chinese, 2,602 English-to-Yi, and 2,649 Yi-to-English items. | | Multiple choice | 15,970 (21.58%) | All 15,970 reference labels point to an available option. In 15,969 items, all option texts are distinct. | | Cloze | 4,925 (6.66%) | All 4,925 prompts have the same character length as their references and preserve every unmasked character. The prompts mask 33.21% of characters on average. | | Translation judgment | 27,384 (37.01%) | 13,776 positive labels (50.31%) and 13,608 negative labels (49.69%). | | Error correction | 5,596 (7.56%) | The reference changes the input text in 3,484 items. Among the 2,112 items with unchanged text, 2,111 explicitly indicate that no correction is needed. | | Character reordering | 3,231 (4.37%) | All 3,231 prompts preserve the reference characters and their counts. In 3,226 items, the character order differs from the reference. | Across the 15,970 multiple-choice items, uniform guessing gives an expected accuracy of **15.67%**, computed using the actual number of options in each item. Always selecting the most frequent answer position, B, gives **15.95%**. This comparison checks one simple answer-position heuristic. A separate check found **73,987 distinct full records** after normalizing Unicode and whitespace, leaving five redundant records in the released NuosuEdu set. The appendix will provide the full quantitative evidence for Nuosu-Benchmark and examples from the expert validation.
AuthorsOpenReview ↗

Response to Reviewer 4Jmn

Q3 | Training ablations and performance declines

The primary contribution of this paper is the construction of Nuosu-Datasets and Nuosu-Benchmark. BIMO-8B demonstrates the value of these resources for language adaptation. We report the training-stage comparisons below, followed by the results of Nuosu-8B, our latest checkpoint trained with a revised recipe.

Contribution of CPT and SFT. We now report the base model, CPT-only, SFT-only, and CPT+SFT results. CPT uses a next-token prediction objective on monolingual Yi data mixed with general-purpose text, while SFT introduces cross-lingual instruction supervision. The monolingual Yi materials include Yi-only books, online tutorials written entirely in Yi, reports, and prose. These records lack corresponding Chinese translations in our collection and are therefore used for CPT.

Training configurationNuosu-Benchmark score
Qwen3-8B, before adaptation29.58
CPT-only34.83
SFT-only, LoRA33.25
CPT+SFT, LoRA45.55
SFT-only, full-parameter fine-tuning37.27
CPT+SFT, full-parameter fine-tuning50.11

The CPT-only results below follow the metric order in Table 5. Their unweighted mean is 34.8275, reported as 34.83 points.

ModelBLEU-1BLEU-2BLEU-3BLEU-4CIDErGLEUMETEORNISTROUGEchrFchrF++1/(1+TER)
CPT-only21.1315.2213.9911.64103.0215.2122.67112.8125.3315.6613.3047.95

CPT alone improves the composite score over Qwen3-8B by 5.25 points. Adding CPT before SFT improves the score by 12.30 points with LoRA and 12.84 points with full-parameter fine-tuning. The full-parameter CPT+SFT model also exceeds CPT-only by 15.28 points, showing the complementary contributions of target-language pre-training and cross-lingual instruction tuning.

NuosuSen and training-data coverage. BIMO-8B’s decline from 32.69 to 29.06 on NuosuSen is consistent with task-specific negative transfer. A plausible explanation is a mismatch between the registers emphasized in training and those evaluated by NuosuSen, which includes colloquial expressions and sentence translation. Everyday expressions make up 6.47% of the parallel corpus, compared with 37.58% for classical verse and 23.72% for government reports. The monolingual Yi corpus is also dominated by news, which accounts for 79.54% of its tokens.

For Nuosu-8B, we grouped the Yi CPT data by difficulty and designed a curriculum sampling function that gradually introduced more difficult material. Both vocabulary configurations used the same curriculum and included general-purpose data accounting for 20% of the total training tokens in each stage, calculated separately for CPT and SFT.

Follow-up configurationNuosu-Benchmark scoreGeneral-capability average
Curriculum sampling with an expanded vocabulary57.6660.22
Curriculum sampling with the original vocabulary (Nuosu-8B)59.4868.90

Under this curriculum, retaining the original vocabulary improved the Nuosu-Benchmark score by 1.82 points and the general-capability average by 8.68 points compared with expanding it. Nuosu-8B also scored 54.88 on NuosuSen, exceeding the reported Qwen3-8B score of 32.69. The revised recipe thus reverses the sentence-level regression observed in the submitted run.

General-capability retention. BIMO-8B declined on all five general benchmarks, a pattern consistent with forgetting after adaptation. Nuosu-8B improves on all five in the reported evaluations.

BenchmarkQwen3-8BBIMO-8BNuosu-8B
MMLU87.5074.4081.30
GPQA62.0038.2055.70
LiveBench67.1048.1060.30
MATH-50097.4051.2080.60
AIME’2567.3031.3066.60
Average76.2648.6468.90

The largest gains over BIMO-8B are on AIME’25 (+35.30 points) and MATH-500 (+29.40 points). Nuosu-8B comes within 0.70 points of Qwen3-8B on AIME’25, while MATH-500 remains the largest gap at 16.80 points. Across the five benchmarks, the average gap to Qwen3-8B narrows from 27.62 to 7.36 points. The follow-up recipe therefore mitigates the loss of general capability while improving both sentence-level Yi performance and the overall Nuosu-Benchmark score.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts in addressing my concerns. However, I remain concerned about the unnormalized aggregation of metrics with different numerical scales, as this affects the interpretation and reliability of the reported performance improvements. Therefore, I will maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.