- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the effects of LoRA CPT on low resource languages in LLMs. The paper shows that training on a set of polysynthetic languages, outputs generated in English become shorter, while still performing comparably on a fact summarization task. On GSM8K, the same shortening of generations is observed, though accuracy also drops significantly indicating that the effect is accompanied by forgetting of more complicated tasks. The work also studies where in the model these behavioral shifts stem from and identify a band of intermediate layers where ablation of LoRA parameters has the biggest effect.
Critical strengths
(S1) The finding by which CPT on polysynthetic languages shortens general generation length is interesting.
(S2) The localization of this effect is interesting.
Critical weaknesses
(CW1) I think this paper requires extensive revisions on the presentation and framing of its results. Details below in (W1.1) - (W1.4).
(CW2) The claims require studying further settings and a more careful interpretation and discussion of the results. Details see below, in particular (W3), (W4), (W5), and (W6).
Detailed feedback: In particular, here are some pointers where to start:
(W1.1) It begins with the title "THE LINGUISTIC LONG TAIL RESHAPES LANGUAGE MODELS". You cite work which already establishes this and you provide no other sources or convincing arguments of why this is a novel or interesting insights. It is almost a truism: training on any data changes/"reshapes" a language model! This is not to say your results aren't interesting (they are imo), but you should focus on and be precise with what you actually find: Generation length is shortened by training on a particular set of low resource languages, while general model behavior stays (mostly) intact, plus a mechanistic study of this phenomenon.
(W1.2) "The linguistic long tail", as you also state in your paper, is much more diverse and long than what you explore. You study a very focused set of languages and tasks, your framing reaches far beyond that which, ultimately, weakens your story.
(W1.3) "These findings position the linguistic long tail as a scientific instrument". Again, you study a very specific setting and can answer only a very specific (interesting) question. Your study makes no recommendation and provides no evidence on how and to which problems in understanding or controlling LLM behavior your approach could help. In other words, your claim requires more theoretic and experimental arguments to be valid this broadly.
(W1.4) Your work does not establish convincing arguments for the "broader implications" you discuss in Section 8.3.
(W2) English is not the only high resource language in Qwen-2.5. Does the shortening effect also extend to, e.g., Chinese?
(W3) You do continued pretraining, yet your chosen dataset size, batch size and thus also number of optimization steps is very small and your results seem to hold mostly in the first half of this training regime. Why did you chose your hyper parameters like that and is it possible that your results would change under choosing more realistic parameters? In case this is a misunderstanding on my side: What previous findings/work did you consider when making this choice or did you perform any ablations?
(W4) Prompting further CoT steps degrades GSM8K performance, disagreeing with Jin et al. (2024). As you state yourselves, an instruction tuned model would be more suitable for this. Why not study one then? Given that the number of steps is an especially interesting feature in CoT reasoning, your approach could have been a very promising, slightly weird (in a good way) solution to that. Also, the step reduction only seems to apply to the additionally prompted setting. Why? Instead, the insight of that experiment is questionable given that the performance degrades so much on math and shows us nothing beyond "math CoTs prompted for more steps become shorter on a model that is not optimized or good at math CoT reasoning". Would be easily fixable by swapping in another, instruction tuned checkpoint into your pipeline.
(W5) I am unconvinced by how you interpret Table 7. You report that "not produce a systematic increase in pre-termination EOS probability", yet the table shows that in all but the 75% length level, the probability of EOS increases consistently to three up to 30 times the base value. Further, why does the 75% step, the only outlier in this trend, show such a big increase across settings, even beyond the final probability?
(W6) You claim that your results generalize and are robust beyond Qwen2.5-7B. Yet you only analyze its smaller, same-family version Qwen2.5-3B and the similar sized Gemma-2-9B. This is hardly a good coverage of the current SOTA "across model sizes" (line 393) and in terms of "model family" (line 394). This ties into (W1), but should also be addressed by including further models, e.g. an instruction tuned version, or if resources allow a larger model.
Comments/typos/writing:
-
citations in lines 081, 112 are not rendered correctly
-
line 123: Which effect? Restate if you start a new chapter.
-
Section 4: Why did you choose these languages to train on in particular?
Questions for authors
See above.
6 comments
W6: Model generalization and Robustness
We agree that broader validation is important. The revised Section 7 (Table 10) now evaluates seven models across three families (Qwen, Gemma, Llama), with all three CPT seeds.
| Model | Condition | Seed | Words | Tokens | Reduction | p |
|---|---|---|---|---|---|---|
| Qwen2.5-3B | Base | – | 45.84 | 57.00 | – | – |
| Polysynthetic | 42 | 30.42 | 40.06 | 33.64% | 1.33 × 10⁻³⁷ | |
| Polysynthetic | 123 | 30.73 | 40.37 | 32.96% | 7.91 × 10⁻³⁵ | |
| Polysynthetic | 456 | 29.90 | 39.73 | 34.78% | 2.39 × 10⁻⁴⁰ | |
| Qwen2.5-7B | Base | – | 46.13 | 56.83 | – | – |
| Polysynthetic | 42 | 31.73 | 39.90 | 31.20% | 1.33 × 10⁻⁴¹ | |
| Polysynthetic | 123 | 30.56 | 38.44 | 33.80% | 9.17 × 10⁻⁴⁹ | |
| Polysynthetic | 456 | 35.78 | 44.57 | 22.40% | 2.39 × 10⁻²³ | |
| Qwen3-4B | Base | – | 52.66 | 64.73 | – | – |
| Polysynthetic | 42 | 44.12 | 54.90 | 16.22% | 1.17 × 10⁻¹³ | |
| Polysynthetic | 123 | 46.42 | 57.38 | 11.86% | 5.18 × 10⁻⁸ | |
| Polysynthetic | 456 | 41.83 | 51.98 | 20.57% | 1.61 × 10⁻¹⁷ | |
| Gemma-2-9B | Base | – | 63.46 | 84.20 | – | – |
| Polysynthetic | 42 | 41.86 | 54.65 | 34.03% | 4.94 × 10⁻³⁷ | |
| Polysynthetic | 123 | 40.46 | 51.39 | 36.23% | 1.12 × 10⁻³⁹ | |
| Polysynthetic | 456 | 39.64 | 50.49 | 37.54% | 3.07 × 10⁻⁴¹ | |
| Gemma-3-4B-PT | Base | – | 87.36 | 115.42 | – | – |
| Polysynthetic | 42 | 36.16 | 46.84 | 58.61% | 1.10 × 10⁻¹¹¹ | |
| Polysynthetic | 123 | 36.29 | 46.99 | 58.46% | 4.23 × 10⁻¹¹² | |
| Polysynthetic | 456 | 38.39 | 49.72 | 56.06% | 4.10 × 10⁻¹⁰⁴ | |
| Llama-3-8B | Base | – | 41.87 | 52.46 | – | – |
| Polysynthetic | 42 | 30.18 | 38.16 | 27.91% | 1.10 × 10⁻¹⁸ | |
| Polysynthetic | 123 | 29.72 | 37.57 | 29.02% | 7.02 × 10⁻²⁰ | |
| Polysynthetic | 456 | 29.78 | 37.70 | 28.86% | 1.53 × 10⁻¹⁹ | |
| Llama-3.1-8B | Base | – | 49.32 | 61.67 | – | – |
| Polysynthetic | 42 | 29.33 | 37.48 | 40.54% | 3.28 × 10⁻³⁰ | |
| Polysynthetic | 123 | 30.94 | 39.44 | 37.27% | 4.21 × 10⁻²⁷ | |
| Polysynthetic | 456 | 31.23 | 40.02 | 36.68% | 1.57 × 10⁻²⁶ |
All 21 model–seed configurations show statistically significant reductions (p < 0.001). The original Gemma results excluded truncated generations, obscuring the magnitude of the effect (as noted in the original table caption). In fact, Gemma exhibits the largest reductions, as its base models are particularly verbose. Revised Table 10 includes all 500 examples, while Appendix D analyzes excessive continuation and confirms that shortening persists among paired non-truncated examples. Updated code has also been pushed to the linked repository.
W7. Code and Data Availability
(W7) No code submission. The work of this community is enabled by an open-source culture which allows us to quickly build on and extend previous findings. This includes this very paper you're submitting for review. Besides, some of your claims would be even stronger if the reviewers had access to your code to check how your have implemented the experiments you describe in the paper.
We respectfully clarify that all code, datasets, and generation details have been publicly available through the anonymous GitHub repository linked in our Reproducibility Statement since the original submission. We fully share the reviewer's commitment to open-source research and reproducibility, which is precisely why we made these materials available from the outset.
Additional Comments
We have corrected the citation rendering in lines 081 and 112 and clarified the wording in the original line 123. The languages were selected for their linguistic properties and data availability, as discussed in Section 4 and Appendix A.
the sources you cite in line 044, do not seem to be relevant for the claim "lower target-language perplexity does not imply better downstream task performance". I did not check every source you cite, but I would carefully check and revise citations for the next revision.
Regarding the citation for 044, there is no citation on line 044 (or in the surrounding lines) of our original submission. More importantly, the quoted claim does not appear anywhere in our manuscript, nor do we discuss target-language perplexity. We are concerned that this comment may have been intended for a different submission or arose from an inaccurate characterization of our work, and would appreciate clarification.
We hope that our substantially expanded experiments, more in-depth analyses, and clarifications have addressed your concerns. If so, we would greatly appreciate your reconsideration of the overall score. Thank you again for your time and detailed feedback!
Thank you for your replies. After carefully considering your rebuttal, I am not convinced this would turn into a paper that passes the bar for ICLR in the next revision. Rather, I recommend to carefully revise your manuscript, perhaps extending on the points other reviewers and me have highlighted and then to resubmit. I appreciate your hard work for the rebuttal though and acknowledge that some of my points have been addressed (W6) or clarified (W7, comment 3). I have adapted my original review to correct the latter points.
Extending, I wish to comment on the following:
We chose The Linguistic Long Tail Reshapes Language Models to reflect a broader scientific vision: overlooked languages may reveal unexpected ways to reshape foundation models; English concision through polysynthetic CPT is our first observed phenomenon.
A title is not a statement of ambition, but should be a transparent representation of what the paper contains. This comment entirely misses the points about framing and transparency I was making above. I would advise you to move such discussions of ambitions and impact to the Discussion section, where it would traditionally be a welcome addition.
With over 7,000 languages worldwide, no single paper can explore them all,
And I have not claimed otherwise.
W3. CPT Hyperparameters and Training Duration
I do not find this answer very helpful.
W4. Base Models, Instruction Tuning, and Mathematical Reasoning
This reply also misses the point I was making.
W5. EOS Probability Interpretation
I welcome this discussion, but I think you need to dig deeper on this instead of restating your papers false claims.
W6: Model generalization and Robustness
I welcome these new results. Perhaps reporting seeds is not the best way to present them. I'd rather accumulate over seeds and report standard deviation.
We respectfully clarify that all code, datasets, and generation details have been publicly available through the anonymous GitHub repository linked in our Reproducibility Statement since the original submission.
Thanks, I missed the link because it is hidden behind the text of said statement. Sorry for the confusion. I would rather present the complete link as footnote.
egarding the citation for 044, there is no citation on line 044 (or in the surrounding lines) of our original submission. More importantly, the quoted claim does not appear anywhere in our manuscript, nor do we discuss target-language perplexity.
Yes! Apologies again, I must have had multiple tabs open and copied this from my review from another paper. Thanks for spotting this.
We thank Reviewer Edv8 for their detailed comments, and for recognizing the novelty and interest of our findings. We have taken your concerns seriously and uploaded a substantially revised manuscript with expanded experiments and deeper analyses. We also hope to clarify two apparent misunderstandings regarding citations and reproducibility. Specifically, the claim attributed to line 044 does not appear in our original manuscript, nor is there a citation on that line or in the surrounding text. Additionally, all code and experimental materials have been available through our linked anonymous repository since the original submission.
W1.1–W1.4. Title, Framing, and Broader Implications
We appreciate your suggestions and are open to reconsidering the title, although we prefer to retain it during rebuttal to avoid confusion. We chose The Linguistic Long Tail Reshapes Language Models to reflect a broader scientific vision: overlooked languages may reveal unexpected ways to reshape foundation models; English concision through polysynthetic CPT is our first observed phenomenon.
With over 7,000 languages worldwide, no single paper can explore them all, especially when even foundation models and multilingual benchmarks cover only a fraction. This is a first demonstration of a new research direction, and we hope this discovery inspires further exploration. Section 8.3 discusses these possibilities as future directions rather than established universal conclusions, and we welcome suggestions for making this distinction clearer.
W2. Chinese Evaluation
We explored Chinese generation (Appendix F), with manual validation by a fluent non-native Chinese speaker and Google Translate cross-checks. The initial five examples showed 10.95% shorter generations on average, but expanded evaluation revealed substantial degradation, including numerical sequences instead of Chinese text. This suggests that our English evaluation settings do not transfer cleanly to Chinese, and further tuning would be needed for a reliable comparison. To support broader, community-led multilingual evaluation, we are considering proposing a shared task at a future *CL conference, following similar initiatives at AmericasNLP and ComputEL.
W3. CPT Hyperparameters and Training Duration
Our small-scale setup is intentional: in low-resource language research, corpora of ~4M tokens are already substantial. To ensure consistency, all training data were drawn exclusively from the Leipzig Corpus and matched in token count and source composition. We also conducted a dose-response analysis (Table 5) and extended CPT to step 500 (Appendix C). Concision improves through step 243, approximately one epoch, but further training causes degradation, with 28.4% of generations reaching the 150-token limit, alongside repetition and language leakage. We therefore selected step 243 to capture the effect before this instability.
W4. Base Models, Instruction Tuning, and Mathematical Reasoning
Our findings do not necessarily contradict Jin et al. (2024), who evaluated closed-weight OpenAI models. We respectfully clarify that we did not suggest instruction-tuned models were more suitable. Rather, we explicitly stated: “This difference may reflect the use of a base model, necessary to isolate the effects of continued pretraining without prior instruction tuning.” CPT on base models, followed by instruction tuning, follows an established training paradigm [1]. Applying CPT directly to instruction-tuned models may also risk degrading previously learned instruction-following capabilities. Our choice of base models is therefore deliberate, allowing us to isolate the effects of linguistic CPT without this additional confound.
Regarding reasoning length, standard CoT may impose a minimum sufficient reasoning length. For example, both models correctly solve GSM8K ID 1208 (Appendix E) in 45 and 44 tokens, respectively. Under “More Steps,” both remain correct, but Base expands to 272 tokens while Polysynthetic uses only 51. Thus, the concision effect becomes particularly visible under explicit pressure to elaborate. We acknowledge the decline in mathematical accuracy and do not claim improved reasoning performance.
[1] The Llama 3 Herd of Models. Meta, 2024
W5. EOS Probability Interpretation
We acknowledge that Polysynthetic CPT increases pre-termination EOS probability at several positions. However, the absolute probabilities remain extremely small, and the pattern is not consistent across positions. At 75%, Polysynthetic exhibits lower EOS probability (0.000226) than Base (0.001276), despite producing shorter responses. The spike at 75% across conditions warrants further investigation, potentially reflecting differences in response structure or token alignment. We have reframed our writing: EOS behavior may contribute to concision, but does not fully explain the observed effect.