- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates cross-lingual transfer through Standardized Yi (Nuosu), a language spoken by millions of people in southwestern China but largely absent from existing LLM training and evaluation ecosystems. The data are collected from authoritative web and expert-provided sources through OCR-based digitization, multi-stage filtering, and expert validation. The authors build a benchmark, Nuosu-Benchmark, containing 110,513 instances across five sub-benchmarks that evaluate lexical knowledge, sentence-level understanding, culturally grounded knowledge, administrative language, and educational comprehension using 12 complementary metrics. This paper evaluates 14 representative LLMs and reveals a substantial gap between general multilingual capability and competence in Standardized Yi.
Critical strengths
- The paper investigated an underrepresented language and released a dataset, a benchmark, and adapted models.
- The paper used a series of data engineering tools and models to curate the dataset, with details reported.
- The paper benchmarked a list of representative models and reported a few metrics across tasks.
Critical weaknesses
- The statement "largely absent from existing LLM training and evaluation ecosystems" is a good selling point. But the corpus has a category called "Official documents". There might be a chance that those official documents have already been explored by existing studies and model training (especially multilingual LLMs adapted for low-resource languages).
- Following the "official documents", both CPT corpus and the benchmark contain the Yi editions of People's Daily. So there is possible train-test contamination between training and benchmarking. This should be addressed.
- There are no ablations in model adaptation; some missing controls are, e.g., CPT-only versus SFT-only versus CPT+SFT.
- The expert validation is not detailed, missing annotator profiles and qualifications, and validation protocols. Translated data and synthetic or extracted data should be better audited at each stage.
Questions for authors
- Is each pivot-augmented Yi–Zh–En sample a separate unit in the split, or a field added to an existing Yi–Zh unit?
- What did the "expert validation" consist of? Can you provide details?
- Why does BIMO-8B score below Qwen3-8B on NuosuSen?
3 comments
Response to Reviewer 9jJa
Q5. Training ablations
The primary contribution of this paper is the construction of Nuosu-Datasets and Nuosu-Benchmark. BIMO-8B demonstrates the value of these resources for language adaptation. We report the training-stage comparisons below, followed by the results of Nuosu-8B, our latest checkpoint trained with a revised recipe.
Contribution of CPT and SFT. We now report the base model, CPT-only, SFT-only, and CPT+SFT results. CPT uses a next-token prediction objective on monolingual Yi data mixed with general-purpose text, while SFT introduces cross-lingual instruction supervision. The monolingual Yi materials include Yi-only books, online tutorials written entirely in Yi, reports, and prose. These records lack corresponding Chinese translations in our collection and are therefore used for CPT.
| Training configuration | Nuosu-Benchmark score |
|---|---|
| Qwen3-8B, before adaptation | 29.58 |
| CPT-only | 34.83 |
| SFT-only, LoRA | 33.25 |
| CPT+SFT, LoRA | 45.55 |
| SFT-only, full-parameter fine-tuning | 37.27 |
| CPT+SFT, full-parameter fine-tuning | 50.11 |
The CPT-only results below follow the metric order in Table 5. Their unweighted mean is 34.8275, reported as 34.83 points.
| Model | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | CIDEr | GLEU | METEOR | NIST | ROUGE | chrF | chrF++ | 1/(1+TER) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| CPT-only | 21.13 | 15.22 | 13.99 | 11.64 | 103.02 | 15.21 | 22.67 | 112.81 | 25.33 | 15.66 | 13.30 | 47.95 |
CPT alone improves the composite score over Qwen3-8B by 5.25 points. Adding CPT before SFT improves the score by 12.30 points with LoRA and 12.84 points with full-parameter fine-tuning. The full-parameter CPT+SFT model also exceeds CPT-only by 15.28 points, showing the complementary contributions of target-language pre-training and cross-lingual instruction tuning.
For Nuosu-8B, we grouped the Yi CPT data by difficulty and designed a curriculum sampling function that gradually introduced more difficult material. Both vocabulary configurations used the same curriculum and included general-purpose data accounting for 20% of the total training tokens in each stage, calculated separately for CPT and SFT.
| Follow-up configuration | Nuosu-Benchmark score | General-capability average |
|---|---|---|
| Curriculum sampling with an expanded vocabulary | 57.66 | 60.22 |
| Curriculum sampling with the original vocabulary (Nuosu-8B) | 59.48 | 68.90 |
Under this curriculum, retaining the original vocabulary improved the Nuosu-Benchmark score by 1.82 points and the general-capability average by 8.68 points compared with expanding it. Nuosu-8B also scored 54.88 on NuosuSen, exceeding the reported Qwen3-8B score of 32.69. The revised recipe thus reverses the sentence-level regression observed in the submitted run.
General-capability retention. BIMO-8B declined on all five general benchmarks, a pattern consistent with forgetting after adaptation. Nuosu-8B improves on all five in the reported evaluations.
| Benchmark | Qwen3-8B | BIMO-8B | Nuosu-8B |
|---|---|---|---|
| MMLU | 87.50 | 74.40 | 81.30 |
| GPQA | 62.00 | 38.20 | 55.70 |
| LiveBench | 67.10 | 48.10 | 60.30 |
| MATH-500 | 97.40 | 51.20 | 80.60 |
| AIME’25 | 67.30 | 31.30 | 66.60 |
| Average | 76.26 | 48.64 | 68.90 |
The largest gains over BIMO-8B are on AIME’25 (+35.30 points) and MATH-500 (+29.40 points). Nuosu-8B comes within 0.70 points of Qwen3-8B on AIME’25, while MATH-500 remains the largest gap at 16.80 points. Across the five benchmarks, the average gap to Qwen3-8B narrows from 27.62 to 7.36 points. The follow-up recipe therefore mitigates the loss of general capability while improving both sentence-level Yi performance and the overall Nuosu-Benchmark score.
Response to Reviewer 9jJa
Q6. Insufficient benchmark validation
We will describe the checks applied at each data-processing stage and the final human review in the main paper.
| Stage | Checking procedure |
|---|---|
| OCR extraction | Two Yi studies experts proofread the recognized text against the scanned source documents. |
| Pivot translation | English was generated automatically from the Chinese side of the original Yi–Chinese record. The augmented records underwent automatic checks within the processing pipeline and retained their original source-unit assignments. |
| TIEA generation | Tasks were produced through programmed templates and transformations and checked automatically. The checks covered available answer labels, preservation of unmasked characters, and preservation of character counts in reordering tasks. |
| Final benchmark review | Two doctoral students from SWUFE first checked item formatting and content against the cross-lingual source data. Two Yi studies experts from SWUN then jointly reviewed the items for factual and linguistic consistency and the skill elicited by each task.The expert review used a joint consensus mechanism: uncertain cases were discussed until the experts agreed on one judgment per item. |
Taking NuosuEdu as an example:
Quantitative evidence.
| Task | Items (% of NuosuEdu) | Findings |
|---|---|---|
| Translation | 16,886 (22.82%) | 5,709 Chinese-to-Yi, 5,926 Yi-to-Chinese, 2,602 English-to-Yi, and 2,649 Yi-to-English items. |
| Multiple choice | 15,970 (21.58%) | All 15,970 reference labels point to an available option. In 15,969 items, all option texts are distinct. |
| Cloze | 4,925 (6.66%) | All 4,925 prompts have the same character length as their references and preserve every unmasked character. The prompts mask 33.21% of characters on average. |
| Translation judgment | 27,384 (37.01%) | 13,776 positive labels (50.31%) and 13,608 negative labels (49.69%). |
| Error correction | 5,596 (7.56%) | The reference changes the input text in 3,484 items. Among the 2,112 items with unchanged text, 2,111 explicitly indicate that no correction is needed. |
| Character reordering | 3,231 (4.37%) | All 3,231 prompts preserve the reference characters and their counts. In 3,226 items, the character order differs from the reference. |
Across the 15,970 multiple-choice items, uniform guessing gives an expected accuracy of 15.67%, computed using the actual number of options in each item. Always selecting the most frequent answer position, B, gives 15.95%. This comparison checks one simple answer-position heuristic. A separate check found 73,987 distinct full records after normalizing Unicode and whitespace, leaving five redundant records in the released NuosuEdu set.
validation examples. Translation and multiple-choice items ask models to produce or recognize cross-lingual equivalents of words and fixed expressions. Judgment items ask whether a proposed translation preserves the source meaning. Cloze, correction, and character-reordering items ask models to recover a target expression under explicit constraints. For example, the experts confirmed 祭后酿恶果, as the Chinese translation of ꄰꉩꀋꏜꉩ,, an item drawn from 《中国彝文典籍译丛》. A reviewed cloze item asks the model to complete _ꇑ_ꀕꃅ_ using a Chinese gloss. Its reference, ꃅꇑꋍꀕꃅ。, preserves the visible characters and supplies the missing ones.
The appendix will provide the full quantitative evidence for Nuosu-Benchmark and examples from the expert validation.
Response to Reviewer 9jJa
Thank you for your valuable comments.
Q1. Official documents and train-test contamination
We clarify the roles of monolingual and bilingual sources, and the contribution of organizing publicly available materials into training and evaluation resources.
CPT sources and their role. Our CPT corpus consists primarily of monolingual Standardized Yi materials, including Yi-only books, online tutorials written entirely in Yi, reports, prose, and other web texts. These records lack corresponding Chinese translations in our collection and are therefore used for next-token pre-training to develop Yi representations. The bilingual collections provide the aligned source material for cross-lingual SFT and benchmark construction. NuosuGov draws primarily on official bilingual materials provided by Southwest Minzu University for public use, together with bilingual material from People’s Daily. Bilingual records assigned to SFT and evaluation follow disjoint underlying source-unit allocations, and all derivatives inherit their source unit’s allocation.
Public documents and prior model exposure. Publicly accessible Yi documents may have been included in the pre-training corpora of models such as Qwen. Our evaluation nevertheless reveals extremely weak practical Yi competence in Qwen3-8B: its Nuosu-Benchmark score is 29.58, below the 31.95 average across the 14 evaluated models. The gap between potential exposure and measured competence motivates systematic language-specific resources. Our contribution is to organize scattered materials into standardized CPT and SFT corpora, construct tasks across five linguistic and usage levels, and establish a unified benchmark with reproducible evaluation. These resources make the collected materials usable for language adaptation and systematic capability assessment.
Q2. Partitioned units of pivot-augmented samples
An augmented Yi–Zh–En example is stored and counted as an additional record, with the English text derived from the Chinese side of its original Yi–Zh pair. The underlying Yi–Zh source unit is the unit of assignment. The original pair and its augmented record share the same split, as do any instruction examples subsequently generated from that source. We assign the records to the training and evaluation pools before applying TIEA. This means an original Yi–Zh pair cannot be assigned to SFT while its Yi–Zh–En version is assigned to evaluation.
Q3. Why does BIMO-8B score below Qwen3-8B on NuosuSen
BIMO-8B scored 29.06 on NuosuSen, 3.63 points below Qwen3-8B. A plausible explanation is a mismatch in register. NuosuSen includes sentence translation and colloquial expressions, while everyday expressions account for only 6.47% of the parallel corpus. Classical verse and government reports account for 37.58% and 23.72%, respectively. The monolingual Yi corpus used for CPT is also dominated by news, which accounts for 79.54% of its tokens. These distributions show that the training data provide much more exposure to formal and literary registers than to everyday expressions.
In the follow-up study, we grouped the Yi CPT data by difficulty, introduced curriculum sampling, retained the original vocabulary, and included general-purpose data at 20% of the total tokens in each training stage. The resulting Nuosu-8B scored 54.88 on NuosuSen, above both BIMO-8B and Qwen3-8B in the reported evaluations. The follow-up shows that sentence-level performance can improve substantially under a revised training recipe. The corpus distribution makes limited coverage of everyday expressions a plausible contributor to the original regression.
Q4. What did the "expert validation" consist of? Can you provide details?
At an earlier stage of processing, two Yi studies experts proofread the OCR outputs against the source scans. For each sub-benchmark, two Yi studies experts then jointly reviewed the items for factual and linguistic consistency and assessed whether answering each item required the skill it was designed to assess. They discussed uncertain cases until they reached consensus and recorded one joint judgment per item.
Two NuosuEdu items illustrate how the experts applied these criteria. For a translation item drawn from 《中国彝文典籍译丛》, they confirmed that
祭后酿恶果,accurately conveys the meaning ofꄰꉩꀋꏜꉩ,. For a cloze item with the Chinese gloss众神纷纷归将去。, they checked that the referenceꃅꇑꋍꀕꃅ。preserves every visible character in_ꇑ_ꀕꃅ_and matches the gloss. The translation example illustrates how the experts checked semantic fidelity, while the cloze example illustrates how they checked both meaning and adherence to the prompt’s constraints.