acceptodds
Under review as a conference paper at ICLR 2027

The Linguistic Long Tail Reshapes Language Models

Abstract

The linguistic long tail contains languages scarce in data yet rich in linguistic structures underrepresented during foundation-model pretraining. Continued pretraining (CPT) is typically used to expand what foundation models know; we show that learning new linguistic structure can instead reshape behaviors they already possess. CPT on small corpora of polysynthetic languages induces 31.2% shorter English generation despite containing no English adaptation data, while preserving lexical fact coverage; matched analytic-language and English CPT do not reproduce the effect. The shift extends to mathematical reasoning and generalizes across model scales. Causal interventions localize most of the effect to CPT-induced updates in a band of intermediate layers; scaling these updates provides graded control over generation length across tasks. These findings position the linguistic long tail as a scientific instrument for understanding foundation models: long-tail languages can reshape broader model behavior.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
linguistic long tail, continued pretraining, generation length

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Edv8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the effects of LoRA CPT on low resource languages in LLMs. The paper shows that training on a set of polysynthetic languages, outputs generated in English become shorter, while still performing comparably on a fact summarization task. On GSM8K, the same shortening of generations is observed, though accuracy also drops significantly indicating that the effect is accompanied by forgetting of more complicated tasks. The work also studies where in the model these behavioral shifts stem from and identify a band of intermediate layers where ablation of LoRA parameters has the biggest effect.

Critical strengths

(S1) The finding by which CPT on polysynthetic languages shortens general generation length is interesting.

(S2) The localization of this effect is interesting.

Critical weaknesses

(CW1) I think this paper requires extensive revisions on the presentation and framing of its results. Details below in (W1.1) - (W1.4).

(CW2) The claims require studying further settings and a more careful interpretation and discussion of the results. Details see below, in particular (W3), (W4), (W5), and (W6).

Detailed feedback: In particular, here are some pointers where to start:

(W1.1) It begins with the title "THE LINGUISTIC LONG TAIL RESHAPES LANGUAGE MODELS". You cite work which already establishes this and you provide no other sources or convincing arguments of why this is a novel or interesting insights. It is almost a truism: training on any data changes/"reshapes" a language model! This is not to say your results aren't interesting (they are imo), but you should focus on and be precise with what you actually find: Generation length is shortened by training on a particular set of low resource languages, while general model behavior stays (mostly) intact, plus a mechanistic study of this phenomenon.

(W1.2) "The linguistic long tail", as you also state in your paper, is much more diverse and long than what you explore. You study a very focused set of languages and tasks, your framing reaches far beyond that which, ultimately, weakens your story.

(W1.3) "These findings position the linguistic long tail as a scientific instrument". Again, you study a very specific setting and can answer only a very specific (interesting) question. Your study makes no recommendation and provides no evidence on how and to which problems in understanding or controlling LLM behavior your approach could help. In other words, your claim requires more theoretic and experimental arguments to be valid this broadly.

(W1.4) Your work does not establish convincing arguments for the "broader implications" you discuss in Section 8.3.

(W2) English is not the only high resource language in Qwen-2.5. Does the shortening effect also extend to, e.g., Chinese?

(W3) You do continued pretraining, yet your chosen dataset size, batch size and thus also number of optimization steps is very small and your results seem to hold mostly in the first half of this training regime. Why did you chose your hyper parameters like that and is it possible that your results would change under choosing more realistic parameters? In case this is a misunderstanding on my side: What previous findings/work did you consider when making this choice or did you perform any ablations?

(W4) Prompting further CoT steps degrades GSM8K performance, disagreeing with Jin et al. (2024). As you state yourselves, an instruction tuned model would be more suitable for this. Why not study one then? Given that the number of steps is an especially interesting feature in CoT reasoning, your approach could have been a very promising, slightly weird (in a good way) solution to that. Also, the step reduction only seems to apply to the additionally prompted setting. Why? Instead, the insight of that experiment is questionable given that the performance degrades so much on math and shows us nothing beyond "math CoTs prompted for more steps become shorter on a model that is not optimized or good at math CoT reasoning". Would be easily fixable by swapping in another, instruction tuned checkpoint into your pipeline.

(W5) I am unconvinced by how you interpret Table 7. You report that "not produce a systematic increase in pre-termination EOS probability", yet the table shows that in all but the 75% length level, the probability of EOS increases consistently to three up to 30 times the base value. Further, why does the 75% step, the only outlier in this trend, show such a big increase across settings, even beyond the final probability?

(W6) You claim that your results generalize and are robust beyond Qwen2.5-7B. Yet you only analyze its smaller, same-family version Qwen2.5-3B and the similar sized Gemma-2-9B. This is hardly a good coverage of the current SOTA "across model sizes" (line 393) and in terms of "model family" (line 394). This ties into (W1), but should also be addressed by including further models, e.g. an instruction tuned version, or if resources allow a larger model.

Comments/typos/writing:

  • citations in lines 081, 112 are not rendered correctly

  • line 123: Which effect? Restate if you start a new chapter.

  • Section 4: Why did you choose these languages to train on in particular?

Questions for authors

See above.

6 comments

We thank Reviewer Edv8 for their detailed comments, and for recognizing the novelty and interest of our findings. We have taken your concerns seriously and uploaded a substantially revised manuscript with expanded experiments and deeper analyses. We also hope to clarify two apparent misunderstandings regarding citations and reproducibility. Specifically, the claim attributed to line 044 does not appear in our original manuscript, nor is there a citation on that line or in the surrounding text. Additionally, all code and experimental materials have been available through our linked anonymous repository since the original submission.

W1.1–W1.4. Title, Framing, and Broader Implications

We appreciate your suggestions and are open to reconsidering the title, although we prefer to retain it during rebuttal to avoid confusion. We chose The Linguistic Long Tail Reshapes Language Models to reflect a broader scientific vision: overlooked languages may reveal unexpected ways to reshape foundation models; English concision through polysynthetic CPT is our first observed phenomenon.

With over 7,000 languages worldwide, no single paper can explore them all, especially when even foundation models and multilingual benchmarks cover only a fraction. This is a first demonstration of a new research direction, and we hope this discovery inspires further exploration. Section 8.3 discusses these possibilities as future directions rather than established universal conclusions, and we welcome suggestions for making this distinction clearer.

W2. Chinese Evaluation

We explored Chinese generation (Appendix F), with manual validation by a fluent non-native Chinese speaker and Google Translate cross-checks. The initial five examples showed 10.95% shorter generations on average, but expanded evaluation revealed substantial degradation, including numerical sequences instead of Chinese text. This suggests that our English evaluation settings do not transfer cleanly to Chinese, and further tuning would be needed for a reliable comparison. To support broader, community-led multilingual evaluation, we are considering proposing a shared task at a future *CL conference, following similar initiatives at AmericasNLP and ComputEL.

W3. CPT Hyperparameters and Training Duration

Our small-scale setup is intentional: in low-resource language research, corpora of ~4M tokens are already substantial. To ensure consistency, all training data were drawn exclusively from the Leipzig Corpus and matched in token count and source composition. We also conducted a dose-response analysis (Table 5) and extended CPT to step 500 (Appendix C). Concision improves through step 243, approximately one epoch, but further training causes degradation, with 28.4% of generations reaching the 150-token limit, alongside repetition and language leakage. We therefore selected step 243 to capture the effect before this instability.

W4. Base Models, Instruction Tuning, and Mathematical Reasoning

Our findings do not necessarily contradict Jin et al. (2024), who evaluated closed-weight OpenAI models. We respectfully clarify that we did not suggest instruction-tuned models were more suitable. Rather, we explicitly stated: “This difference may reflect the use of a base model, necessary to isolate the effects of continued pretraining without prior instruction tuning.” CPT on base models, followed by instruction tuning, follows an established training paradigm [1]. Applying CPT directly to instruction-tuned models may also risk degrading previously learned instruction-following capabilities. Our choice of base models is therefore deliberate, allowing us to isolate the effects of linguistic CPT without this additional confound.

Regarding reasoning length, standard CoT may impose a minimum sufficient reasoning length. For example, both models correctly solve GSM8K ID 1208 (Appendix E) in 45 and 44 tokens, respectively. Under “More Steps,” both remain correct, but Base expands to 272 tokens while Polysynthetic uses only 51. Thus, the concision effect becomes particularly visible under explicit pressure to elaborate. We acknowledge the decline in mathematical accuracy and do not claim improved reasoning performance.

[1] The Llama 3 Herd of Models. Meta, 2024

W5. EOS Probability Interpretation

We acknowledge that Polysynthetic CPT increases pre-termination EOS probability at several positions. However, the absolute probabilities remain extremely small, and the pattern is not consistent across positions. At 75%, Polysynthetic exhibits lower EOS probability (0.000226) than Base (0.001276), despite producing shorter responses. The spike at 75% across conditions warrants further investigation, potentially reflecting differences in response structure or token alignment. We have reframed our writing: EOS behavior may contribute to concision, but does not fully explain the observed effect.

W6: Model generalization and Robustness

We agree that broader validation is important. The revised Section 7 (Table 10) now evaluates seven models across three families (Qwen, Gemma, Llama), with all three CPT seeds.

ModelConditionSeedWordsTokensReductionp
Qwen2.5-3BBase–45.8457.00––
Polysynthetic4230.4240.0633.64%1.33 × 10⁻³⁷
Polysynthetic12330.7340.3732.96%7.91 × 10⁻³⁵
Polysynthetic45629.9039.7334.78%2.39 × 10⁻⁴⁰
Qwen2.5-7BBase–46.1356.83––
Polysynthetic4231.7339.9031.20%1.33 × 10⁻⁴¹
Polysynthetic12330.5638.4433.80%9.17 × 10⁻⁴⁹
Polysynthetic45635.7844.5722.40%2.39 × 10⁻²³
Qwen3-4BBase–52.6664.73––
Polysynthetic4244.1254.9016.22%1.17 × 10⁻¹³
Polysynthetic12346.4257.3811.86%5.18 × 10⁻⁸
Polysynthetic45641.8351.9820.57%1.61 × 10⁻¹⁷
Gemma-2-9BBase–63.4684.20––
Polysynthetic4241.8654.6534.03%4.94 × 10⁻³⁷
Polysynthetic12340.4651.3936.23%1.12 × 10⁻³⁹
Polysynthetic45639.6450.4937.54%3.07 × 10⁻⁴¹
Gemma-3-4B-PTBase–87.36115.42––
Polysynthetic4236.1646.8458.61%1.10 × 10⁻¹¹¹
Polysynthetic12336.2946.9958.46%4.23 × 10⁻¹¹²
Polysynthetic45638.3949.7256.06%4.10 × 10⁻¹⁰⁴
Llama-3-8BBase–41.8752.46––
Polysynthetic4230.1838.1627.91%1.10 × 10⁻¹⁸
Polysynthetic12329.7237.5729.02%7.02 × 10⁻²⁰
Polysynthetic45629.7837.7028.86%1.53 × 10⁻¹⁹
Llama-3.1-8BBase–49.3261.67––
Polysynthetic4229.3337.4840.54%3.28 × 10⁻³⁰
Polysynthetic12330.9439.4437.27%4.21 × 10⁻²⁷
Polysynthetic45631.2340.0236.68%1.57 × 10⁻²⁶

All 21 model–seed configurations show statistically significant reductions (p < 0.001). The original Gemma results excluded truncated generations, obscuring the magnitude of the effect (as noted in the original table caption). In fact, Gemma exhibits the largest reductions, as its base models are particularly verbose. Revised Table 10 includes all 500 examples, while Appendix D analyzes excessive continuation and confirms that shortening persists among paired non-truncated examples. Updated code has also been pushed to the linked repository.

W7. Code and Data Availability

(W7) No code submission. The work of this community is enabled by an open-source culture which allows us to quickly build on and extend previous findings. This includes this very paper you're submitting for review. Besides, some of your claims would be even stronger if the reviewers had access to your code to check how your have implemented the experiments you describe in the paper.

We respectfully clarify that all code, datasets, and generation details have been publicly available through the anonymous GitHub repository linked in our Reproducibility Statement since the original submission. We fully share the reviewer's commitment to open-source research and reproducibility, which is precisely why we made these materials available from the outset.

Additional Comments

We have corrected the citation rendering in lines 081 and 112 and clarified the wording in the original line 123. The languages were selected for their linguistic properties and data availability, as discussed in Section 4 and Appendix A.

the sources you cite in line 044, do not seem to be relevant for the claim "lower target-language perplexity does not imply better downstream task performance". I did not check every source you cite, but I would carefully check and revise citations for the next revision.

Regarding the citation for 044, there is no citation on line 044 (or in the surrounding lines) of our original submission. More importantly, the quoted claim does not appear anywhere in our manuscript, nor do we discuss target-language perplexity. We are concerned that this comment may have been intended for a different submission or arose from an inaccurate characterization of our work, and would appreciate clarification.

We hope that our substantially expanded experiments, more in-depth analyses, and clarifications have addressed your concerns. If so, we would greatly appreciate your reconsideration of the overall score. Thank you again for your time and detailed feedback!

Thank you for your replies. After carefully considering your rebuttal, I am not convinced this would turn into a paper that passes the bar for ICLR in the next revision. Rather, I recommend to carefully revise your manuscript, perhaps extending on the points other reviewers and me have highlighted and then to resubmit. I appreciate your hard work for the rebuttal though and acknowledge that some of my points have been addressed (W6) or clarified (W7, comment 3). I have adapted my original review to correct the latter points.

Extending, I wish to comment on the following:

We chose The Linguistic Long Tail Reshapes Language Models to reflect a broader scientific vision: overlooked languages may reveal unexpected ways to reshape foundation models; English concision through polysynthetic CPT is our first observed phenomenon.

A title is not a statement of ambition, but should be a transparent representation of what the paper contains. This comment entirely misses the points about framing and transparency I was making above. I would advise you to move such discussions of ambitions and impact to the Discussion section, where it would traditionally be a welcome addition.

With over 7,000 languages worldwide, no single paper can explore them all,

And I have not claimed otherwise.

W3. CPT Hyperparameters and Training Duration

I do not find this answer very helpful.

W4. Base Models, Instruction Tuning, and Mathematical Reasoning

This reply also misses the point I was making.

W5. EOS Probability Interpretation

I welcome this discussion, but I think you need to dig deeper on this instead of restating your papers false claims.

W6: Model generalization and Robustness

I welcome these new results. Perhaps reporting seeds is not the best way to present them. I'd rather accumulate over seeds and report standard deviation.

We respectfully clarify that all code, datasets, and generation details have been publicly available through the anonymous GitHub repository linked in our Reproducibility Statement since the original submission.

Thanks, I missed the link because it is hidden behind the text of said statement. Sorry for the confusion. I would rather present the complete link as footnote.

egarding the citation for 044, there is no citation on line 044 (or in the surrounding lines) of our original submission. More importantly, the quoted claim does not appear anywhere in our manuscript, nor do we discuss target-language perplexity.

Yes! Apologies again, I must have had multiple tabs open and copied this from my review from another paper. Thanks for spotting this.

Reviewer_yxLgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper asks whether small-scale LoRA continued pretraining (CPT) on low-resource polysynthetic languages (Navajo, Kalaallisut) can change the English generation behavior a model already has. The authors report that, with no English adaptation data, the polysynthetic-CPT model produces shorter English outputs on a controlled data-to-text task while fact recall and information density are maintained, and that analytic-language CPT and English CPT controls matched in token count and source do not show the same change. On GSM8K, generation length is similar across conditions under standard CoT; a difference appears only under an explicit prompt to produce more reasoning steps, where the polysynthetic model expands less, though its accuracy is also lower. The mechanism analysis argues against an early-termination (EOS) explanation and uses layer-block ablation and scaling to attribute most of the effect to updates in layers 7–13, which also affect reasoning length on GSM8K. Additional experiments on a 3B Qwen model and Gemma-2-9B show the effect in the same direction, although the Gemma effect is considerably smaller. On this basis, the authors propose the linguistic long tail as a scientific instrument for understanding foundation models.

Critical strengths
  • S1. The experimental setup is carefully constructed. The four conditions (Base / Polysynthetic / Analytic / English) are matched in token count and source composition (web, news, Wikipedia), and the English CPT control shows that extra training on the evaluation language itself is not the cause of the effect.
  • S2. On the data-to-text task, the claim is "compression with content preserved," not just shortening, and this is backed by measurement. Reporting length, recall, and density together, and explicitly rejecting the early-termination hypothesis via EOS probability analysis, strengthens the interpretation.
  • S3. The mechanism analysis goes beyond correlation. The layer-block ablation and scaling experiments attribute most of the effect to layers 7–13 and show that generation length can be controlled in a graded way by the size of those updates, and the same intervention also changes reasoning length on GSM8K. Reporting that recall drops sharply at 1.5× over-scaling is also honest.
  • S4. Scale and reproducibility. The experiments run on 4M tokens, LoRA, and four 24GB GPUs, and the code, data, and evaluation scripts are released. The findings are interesting, and the new framing is a conceptual contribution.
Critical weaknesses
  • W1. No discussion of a confounding variable. The polysynthetic languages are almost absent from Qwen's pretraining data, whereas the analytic languages used as the comparison group are medium-resource languages the model has already learned to some degree. The current design cannot distinguish "polysynthesis changes English" from "an unseen low-resource language changes the structure of English output."
  • W2. Too few models. Most experiments rely on a single model, Qwen2.5-7B. The authors claim the effect "generalizes across model scales" and "persists across controls, model scales, random seeds, and model families," but only Gemma-2-9B was tested, and in a limited way, which is not enough to support generalization. Moreover, the shortening effect on Gemma-2-9B is 8.4%, about a quarter of the Qwen effect.
  • W3. No discussion of tokenizer effects. The Qwen tokenizer keeps frequent words as whole tokens but encodes rare words at the byte level. Most of the low-resource language text was likely byte-encoded. Byte-level tokens could therefore be another factor that changes the LLM's output style, but this is not discussed.
  • W4. The "why" is not explained. The paper shows that polysynthetic CPT can change an LLM's output style, but does not explain why. It presents an interesting hypothesis supported only by empirical results; without a theoretical basis, it is hard to generalize.
  • W5. The GSM8K interpretation is vague. Under the "More Steps" setting the output is shorter, but under the standard setting nearly all CPT conditions produce similar lengths. The authors describe this as "suppressing unnecessary expansion," but "unnecessary" is rather abstract.
  • W6. The accuracy drop is not explained. The authors claim polysynthetic CPT reduces tokens while preserving content, but they do not explain why accuracy on GSM8K decreases.
  • W7. Part of the ablation result is unexplained. In Table 9, the polysynthetic model with layers 7–13 turned off actually produces more tokens (194.69), suggesting that the other layers push toward longer output. No analysis of this is provided.
Questions for authors

Please see Weaknesses.

4 comments

We sincerely thank Reviewer yxLg for the detailed and constructive feedback! We are heartened by your recognition of the novelty of our contribution, careful experimental design, mechanistic analysis, and reproducibility through our publicly released code and data. Guided by your suggestions, we have expanded the experiments and strengthened the analyses in our newly uploaded revised manuscript. We address your concerns below:

W1: Confounding variable

We added a low-resource fusional-language control using Galician (Table 2). Unlike Polysynthetic CPT, Galician CPT increases generation length by 14.1% relative to Base (Table 4). To emphasize, low-resource does not mean no-resource: Navajo has ~161,000 U.S. home-language speakers (2021 Census), while Greenland has ~57,000 residents, many of whom speak Kalaallisut. These languages are computationally underrepresented, but cannot be assumed entirely absent from Qwen's pretraining. Although morphological synthesis cannot yet be isolated as the sole cause, our controls support a distinctive effect of Polysynthetic CPT. We are excited to explore what other linguistic structures might reveal! To enable broader, community-led evaluation, we are considering proposing a shared task at a *CL conference, following similar initiatives at AmericasNLP and ComputEL.

W2: Model generalization and Robustness

We agree that broader validation is important. The revised Section 7 (Table 10) now evaluates seven models across three families (Qwen, Gemma, Llama), with all three CPT seeds.

ModelConditionSeedWordsTokensReductionp
Qwen2.5-3BBase–45.8457.00––
Polysynthetic4230.4240.0633.64%1.33 × 10⁻³⁷
Polysynthetic12330.7340.3732.96%7.91 × 10⁻³⁵
Polysynthetic45629.9039.7334.78%2.39 × 10⁻⁴⁰
Qwen2.5-7BBase–46.1356.83––
Polysynthetic4231.7339.9031.20%1.33 × 10⁻⁴¹
Polysynthetic12330.5638.4433.80%9.17 × 10⁻⁴⁹
Polysynthetic45635.7844.5722.40%2.39 × 10⁻²³
Qwen3-4BBase–52.6664.73––
Polysynthetic4244.1254.9016.22%1.17 × 10⁻¹³
Polysynthetic12346.4257.3811.86%5.18 × 10⁻⁸
Polysynthetic45641.8351.9820.57%1.61 × 10⁻¹⁷
Gemma-2-9BBase–63.4684.20––
Polysynthetic4241.8654.6534.03%4.94 × 10⁻³⁷
Polysynthetic12340.4651.3936.23%1.12 × 10⁻³⁹
Polysynthetic45639.6450.4937.54%3.07 × 10⁻⁴¹
Gemma-3-4B-PTBase–87.36115.42––
Polysynthetic4236.1646.8458.61%1.10 × 10⁻¹¹¹
Polysynthetic12336.2946.9958.46%4.23 × 10⁻¹¹²
Polysynthetic45638.3949.7256.06%4.10 × 10⁻¹⁰⁴
Llama-3-8BBase–41.8752.46––
Polysynthetic4230.1838.1627.91%1.10 × 10⁻¹⁸
Polysynthetic12329.7237.5729.02%7.02 × 10⁻²⁰
Polysynthetic45629.7837.7028.86%1.53 × 10⁻¹⁹
Llama-3.1-8BBase–49.3261.67––
Polysynthetic4229.3337.4840.54%3.28 × 10⁻³⁰
Polysynthetic12330.9439.4437.27%4.21 × 10⁻²⁷
Polysynthetic45631.2340.0236.68%1.57 × 10⁻²⁶

All 21 model–seed configurations show statistically significant reductions (p < 0.001). The original Gemma results excluded truncated generations, obscuring the magnitude of the effect (as noted in the original table caption). In fact, Gemma exhibits the largest reductions, as its base models are particularly verbose. Revised Table 10 includes all 500 examples, while Appendix D analyzes excessive continuation and confirms that shortening persists among paired non-truncated examples. Updated code has also been pushed to the linked repository.

W3: Tokenizer Effects

We agree this is a plausible confound and have added a tokenizer fragmentation analysis (Appendix G, Table 19). Polysynthetic CPT has 19.28% single-byte tokens, compared with 17.32% for Analytic CPT, despite their substantially different generation outcomes. Thus, single-byte prevalence alone cannot explain the effect, although other tokenizer-related factors remain possible. We included an acknowledgement of this limitation in Section 8.1.

During this analysis, we corrected a likely typo in Table 2: the Polysynthetic corpus contains 3.93M, not 3.97M tokens. We rechecked and confirmed all other language counts using the Qwen2.5 tokenizer. We wanted to be completely transparent about this, and the training corpus remains unchanged and is available in our repository for independent verification.

W4: Mechanistic Explanation

As this phenomenon is newly identified, our first priority has been establishing its reproducibility across linguistic controls, model families, and random seeds. Nevertheless, Section 6 goes beyond observation: EOS analysis examines early termination, layer ablations localize 89.7% of the concision effect to layers 7–13, and scaling provides causal control over generation length. The same intervention transfers to GSM8K (Table 9). These results identify where and how the learned updates affect generation, although the underlying representational mechanism remains an exciting open question.

W5. GSM8K Interpretation

Similar lengths under standard prompting may reflect a minimum amount of reasoning needed to solve mathematical problems, leaving little room for compression. GSM8K ID 1208 (Appendix E, Table 16) illustrates this: both models answer correctly in approximately 45 tokens. Under "More Steps," Base expands to 272 tokens, while Polysynthetic uses only 51 for the same correct answer. This suggests that Polysynthetic CPT attenuates prompt-induced elaboration rather than necessarily shortening essential reasoning.We have clarified this interpretation in our revision, with a reference to Appendix E.

W6. Accuracy Trade-off

We acknowledge the accuracy trade-off: under standard CoT, Polysynthetic achieves 77.86% accuracy versus 82.26% for Base (Table 6). This suggests that CPT can alter reasoning behavior without necessarily improving mathematical competence. Appendix E (Table 17, GSM8K ID 116) illustrates a related limitation: both models answer correctly under standard prompting but introduce incorrect proportional relationships when prompted to generate more steps, although Polysynthetic is indeed more concise than Base. To separate concision from correctness, we additionally analyze problems answered correctly by all conditions. Under "More Steps," Polysynthetic generates 91.53 tokens versus 124.75 for Base on these shared-correct problems.

W7. Layer Ablation

Thank you for highlighting this observation. Ablating layers 7–13 increases GSM8K generation length to 194.69 tokens, exceeding Base's 148.07 (Tables 6 and 9). However, this does not necessarily imply that the remaining layers independently promote verbosity, since CPT-induced updates across layers may interact non-additively. Restoring layers 7–13 at 0×, 0.5×, and 1× progressively reduces reasoning length from 194.69 to 145.32 to 109.00 tokens (Table 9). This establishes a causal contribution of these updates.

Thank you again for your thoughtful feedback, which has helped us strengthen this paper considerably. We hope the new experiments address your concerns and would be grateful if you might reconsider your score. Above all, we hope our findings have sparked some excitement about the untapped potential of the linguistic long tail!

Thank you for the thorough response. The experimental results are interesting, but my concerns have not yet been sufficiently resolved.

  • I understand that low-resource does not mean no-resource. However, this is only a matter of degree. It is clear that LLMs are much more strongly fitted to high-resource languages such as English and Chinese. The revised version acknowledges this as a limitation, but this is not sufficient.
  • I appreciate the added results for the fusional language. But why does this language actually increase the number of tokens? An analysis or at least a conjecture would be helpful.
  • As shown in Table 4, all non-polysynthetic CPT conditions (Analytic, Fusional, English) produce numerically longer English generations than Base. Only the polysynthetic condition moves in the opposite direction, yet the paper does not address this, and no statistical tests against Base are provided.

Ultimately, my main concern is that, beyond the empirical finding that learning a new language changes output token counts, the paper needs experiments on which property of the new language actually causes this change.

Therefore, I maintain my score.

Dear Reviewer yxLg,

Thank you for your follow-up. We agree that language familiarity and morphological structure cannot yet be fully disentangled. In response to your suggestion, we have revised Section 8.1 to make explicit our conjecture that the high degree of morphological synthesis in polysynthetic languages may drive the distinct concision effect observed after CPT, while acknowledging other possible contributing factors. Our Analytic, Fusional, and English CPT conditions serve as controls to establish the distinctiveness of this effect, rather than as independent phenomena we seek to explain.

We respectfully emphasize that establishing a novel empirical phenomenon and fully explaining its linguistic cause are distinct scientific contributions. Our findings demonstrate that continued pretraining on polysynthetic languages induces a pronounced concision effect in English, spanning data-to-text generation and mathematical reasoning tasks. This phenomenon is replicated across seven models, three families, and three random seeds, with additional support from linguistic controls and mechanistic interventions. Identifying its precise linguistic cause remains an exciting direction for future work.

Thank you for your feedback!

Sincerely,

Authors

Reviewer_N6UPOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores how continued pretraining on very low-resourced polysynthetic languages leads to concision. They focus on Qwen2.5-7B and CPT the model on three types of languages and then evaluate the model on (i) controlled data-to text generation and (ii) CoT reasoning tasks. They find that in most settings, models CPTed on polysynthetic languages generate shorter sequences. They then conduct an ablation study and find that ablating layers 7-13 strongly show the reverse effect.

Critical strengths
  • The paper focuses on long-tail phenomena in model training and the finding is interesting. Importantly, it bridges low-resourced languages for CPT with concise generation and I believe this is new in the field.

  • The writing is clear although I think the organization of the structure can be better. The experiments are also documented very clearly. Every experiment has its motivation well-explained.

Critical weaknesses
  • The evaluation remains limited in tasks, models, and language. This paper only evaluates the models on two tasks, one is data-to-text generation and the other one math reasoning. In the math reasoning task, the concision effect is not even observed in the standard CoT setting. In terms of model families, only two families are tested and Gemma does not show a huge decrease in generation length (although the authors show that such decrease is significant). Since Qwen is multilingual, why is the analysis only limited to English? Is this concision only observed in English? I think the finding would be stronger if such concision effect is more consistent across more tasks, model families, and languages.

  • The motivation of the paper is not clear to me in the introduction. I do not really understand why it is important to study how CPT on low-resource languages affects performance in a dominant language. If authors want to claim that concision is important and their approach is useful, but this approach does negatively influence the reasoning performance (Table 6).

  • The interpretability study is a bit shallow to me. They simply locate what layers are influenced by the low-resourced CPT. I do not see a clear connection with the previous experiments.

Questions for authors
  • Do the authors attribute the shorter sequences observed after CPT on polysynthetic languages to the model’s limited exposure to such data, or to specific linguistic properties of polysynthetic languages?
  • How exactly is the lexical fact recall calculated? In particular, how are ''content-bearing units'' defined and counted? Are they simply content words? If so, the high lexical-fact recall may be partly explained by the tendency of models after polysynthetic-language CPT to reuse words from the input rather than generate novel ones.
  • Section 7, why were Qwen2.5-3B and Gemma-2-9B not evaluated with two additional random seeds as well? Additional seeds would help assess the robustness and statistical reliability of the reported results.
  • Some minor points: (i) some tables can be presented with figures instead (e.g., Table 7, Table 5) (ii) citation format in line 75 (iii) the captions should be above the tables.
8 comments

We thank Reviewer N6UP for their thoughtful review, especially for recognizing the novelty of our work! It is such an exciting discovery: languages in the linguistic long tail, so often overlooked, may hold the key to understanding and reshaping foundation models. Following your suggestions, we have uploaded a revision of our paper with expanded experiments and deeper analyses, and address your comments below.

1. Motivation

Our contribution is not a concision optimization technique, but the discovery that CPT on underrepresented languages can reshape existing model behavior, demonstrated through English generation concision without any English adaptation data, across task-to-text generation and mathematical reasoning. This opens a world of possibilities: perhaps CPT on other languages from the linguistic long tail could induce greater expressiveness or creativity? While concision itself offers real token-cost savings, improving mathematical accuracy is not the objective of this paper. Above all, this paper presents a different research angle; that the value of endangered languages extends far beyond cultural preservation or translation, and they have the power to reshape foundation models.

2. Attribution and Linguistic Controls

We added a low-resource fusional-language control using Galician (Table 2). Unlike Polysynthetic CPT, Galician CPT increases generation length by 14.1% relative to Base (Table 4). To emphasize, low-resource does not mean no-resource: Navajo has ~161,000 U.S. home-language speakers (2021 Census), while Greenland has ~57,000 residents, many of whom speak Kalaallisut. These languages are computationally underrepresented, but cannot be assumed entirely absent from Qwen's pretraining. Although morphological synthesis cannot yet be isolated as the sole cause, our controls support a distinctive effect of Polysynthetic CPT. We are excited to explore what other linguistic structures might reveal! To enable broader, community-led evaluation, we are considering proposing a shared task at a *CL conference, following similar initiatives at AmericasNLP and ComputEL.

3. Model generalization and Robustness

We agree that broader validation is important. The revised Section 7 (Table 10) now evaluates seven models across three families (Qwen, Gemma, Llama), with all three CPT seeds.

ModelConditionSeedWordsTokensReductionp
Qwen2.5-3BBase–45.8457.00––
Polysynthetic4230.4240.0633.64%1.33 × 10⁻³⁷
Polysynthetic12330.7340.3732.96%7.91 × 10⁻³⁵
Polysynthetic45629.9039.7334.78%2.39 × 10⁻⁴⁰
Qwen2.5-7BBase–46.1356.83––
Polysynthetic4231.7339.9031.20%1.33 × 10⁻⁴¹
Polysynthetic12330.5638.4433.80%9.17 × 10⁻⁴⁹
Polysynthetic45635.7844.5722.40%2.39 × 10⁻²³
Qwen3-4BBase–52.6664.73––
Polysynthetic4244.1254.9016.22%1.17 × 10⁻¹³
Polysynthetic12346.4257.3811.86%5.18 × 10⁻⁸
Polysynthetic45641.8351.9820.57%1.61 × 10⁻¹⁷
Gemma-2-9BBase–63.4684.20––
Polysynthetic4241.8654.6534.03%4.94 × 10⁻³⁷
Polysynthetic12340.4651.3936.23%1.12 × 10⁻³⁹
Polysynthetic45639.6450.4937.54%3.07 × 10⁻⁴¹
Gemma-3-4B-PTBase–87.36115.42––
Polysynthetic4236.1646.8458.61%1.10 × 10⁻¹¹¹
Polysynthetic12336.2946.9958.46%4.23 × 10⁻¹¹²
Polysynthetic45638.3949.7256.06%4.10 × 10⁻¹⁰⁴
Llama-3-8BBase–41.8752.46––
Polysynthetic4230.1838.1627.91%1.10 × 10⁻¹⁸
Polysynthetic12329.7237.5729.02%7.02 × 10⁻²⁰
Polysynthetic45629.7837.7028.86%1.53 × 10⁻¹⁹
Llama-3.1-8BBase–49.3261.67––
Polysynthetic4229.3337.4840.54%3.28 × 10⁻³⁰
Polysynthetic12330.9439.4437.27%4.21 × 10⁻²⁷
Polysynthetic45631.2340.0236.68%1.57 × 10⁻²⁶

All 21 model–seed configurations show statistically significant reductions (p < 0.001). The original Gemma results excluded truncated generations, obscuring the magnitude of the effect (as noted in the original table caption). In fact, Gemma exhibits the largest reductions, as its base models are particularly verbose. Revised Table 10 includes all 500 examples, while Appendix D analyzes excessive continuation and confirms that shortening persists among paired non-truncated examples. Updated code has also been pushed to the linked repository.

4. CoT Concision and Reasoning Accuracy

The similar lengths under standard prompting may reflect a minimum number of reasoning steps needed to solve mathematical problems, leaving little room for further compression. GSM8K ID 1208 (Appendix E, Table 16) illustrates this: both Base and Polysynthetic solve the problem correctly in approximately 45 tokens. Under the "More Steps" prompt, Base expands to 272 tokens to reach the same answer, while Polysynthetic uses only 51. This suggests that Polysynthetic CPT attenuates elaboration rather than essential reasoning. Improving mathematical accuracy was not the objective of this experiment, and the observed accuracy trade-off is explicitly reported.

5. Lexical Fact Recall

We calculate recall by lowercasing and removing punctuation, excluding predefined stopwords, and measuring exact lexical overlap between each input fact and the generated response, averaged across facts. As the reviewer notes, this rewards lexical reuse rather than semantic paraphrasing. However, the task is deliberately simple: models transform fixed facts into prose (Table 3), and all conditions achieve high recall (>0.91). The primary difference is how verbosely they express those facts.

6. Chinese Evaluation

As a pilot test, we explored Chinese generation (Appendix F), with manual validation by a fluent non-native Chinese speaker and cross-checking with Google Translate. The initial five examples showed 10.95% shorter generations on average, but the larger evaluation revealed substantial degradation, including outputs consisting of numerical sequences rather than Chinese text. This suggests that our English evaluation settings do not transfer cleanly to Chinese, and further tuning would be needed for a reliable comparison. This is also why we envision a community-led shared task, where native speakers can help ensure that generations are not only in the correct language, but also natural and factually accurate.

7. Interpretability

Section 6 investigates EOS probabilities, localizes the concision effect to layers 7–13, and demonstrates causal control through ablation and scaling (Table 8). The same intervention also transfers to the out-of-domain GSM8K task, reducing reasoning length from 194.69 to 109.00 tokens (Table 9). Given the novelty of this phenomenon, we have prioritized establishing its robustness across models, seeds, and linguistic controls. Deeper interpretability remains an exciting direction for future work.

8. Presentation

Citation formatting and table-caption placement have been fixed. Regarding Tables 5 and 7, we were thinking maybe a line plot? Would you have any suggestions for figure format? Thanks!

Once again, we sincerely appreciate the time and effort you put into reviewing our paper, and wanted to honor your feedback by working hard to address the concerns. Since the reviews were released, we have been running experiments nonstop, particularly the model generalization experiments, and are delighted to have completed them in time to share with you. Your suggestions have directly helped us produce a stronger paper! If you feel we have addressed your concerns, we would be very grateful if you would consider revisiting your score. Thank you again for reading our work, and we hope it has sparked some excitement about the possibilities hidden in the linguistic long tail.

Thank you for your response! Yes, the updated results make the paper look a lot stronger. Given the amount of work to be added, I think it would be better to just submit the paper in the next cycle or to another conference.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank You and Looking Ahead

Dear Reviewers,

As the rebuttal period concludes, we wanted to thank you all for reading and engaging with our work. Guided by your feedback, we have sharpened our framing, added a low-resource Galician CPT control, demonstrated robust generalization across seven models from three families and three random seeds, and deepened our GSM8K, tokenizer, and mechanistic analyses. All these changes have already been incorporated into our latest revised manuscript, available for download. Your suggestions have directly helped make this paper stronger.

We are especially heartened that all three reviewers recognized the novelty and interest of our findings: continued pretraining on low-resource polysynthetic languages (Navajo & Kalaallisut) can unexpectedly induce more concise English generation. In the spirit of ICLR's commitment to foundational research and interdisciplinary discovery, we hope our findings inspire a new research direction, broader multilingual investigations, or even a community-wide shared task exploring the untapped potential of long-tail languages. Perhaps the next breakthrough in machine learning will come from the linguistic diversity we have yet to explore.

Warmly,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.