acceptodds
Under review as a conference paper at ICLR 2027

SPELL-OUT: Preserving Semantic Progression in Cross-Lingual Alignment

Abstract

Cross-Lingual Alignment (CLA) of LLMs has sought to improve multilingual consistency by aligning representations across languages while preserving general capabilities. However, it remains underexplored how to preserve shared semantic progression without collapsing language-specific realization or undermining general capabilities. In this work, we hypothesize that cross-lingual trajectories are better characterized as language-specific realizations of a shared semantic progression, and that CLA should preserve this structure rather than collapse it. We introduce SPELL-OUT, a relational representation alignment objective that uses the source-side trajectory to anchor semantic progression while adapting the target-side realization trajectory. We evaluate SPELL-OUT across diverse multilingual settings and general-capability evaluations. Representation-level analyses further characterize how shared semantic progress is preserved while language-specific trajectory structure remains differentiated. Further analyses show that SPELL-OUT supports diverse target languages under high-resource-language anchors, suggesting broader applicability beyond individual language pairs.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
multilingual language models, cross-lingual alignment, representation learning, semantic trajectories, language-specific representations

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_i7fCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SPELL-OUT, a cross-lingual alignment method that aims to preserve shared semantic progression while allowing language-specific representations to remain distinct. The method aligns teacher-forced prediction-residual trajectories across parallel languages and introduces a source-side preservation objective to limit representation drift. Experiments across multiple multilingual models and language pairs evaluate target-language adaptation and source-language retention. The authors further analyze representation trajectories, ordering correspondence, and sensitivity to source languages and training objectives.

Critical strengths
  1. The paper addresses a meaningful problem in cross-lingual alignment. Directly aligning representations across languages may overlook differences in word order, grammatical structure, and linguistic expression. The idea of preserving shared semantic progression while allowing language-specific realizations is a reasonable and interesting direction.

  2. The proposed method is clearly formulated and accompanied by several analyses. SPELL-OUT combines teacher-forced residual trajectories, source-anchored relational alignment, and explicit source preservation. The authors also examine representation geometry, ordering correspondence, source-language sensitivity, and objective-weight sensitivity, rather than relying solely on downstream performance.

Critical weaknesses
  1. The empirical improvements are marginal, and their statistical significance is unclear. The gains are also relatively small across individual models. It is helpful to establish whether the improvements are consistent across random seeds and statistically significant. Without such evidence, the practical effectiveness of the proposed method remains unclear.

  2. The connection between the proposed objective and semantic progression is not sufficiently established. SPELL-OUT aligns prediction-residual trajectories, but it is unclear why these residuals should reliably capture semantic progression rather than token prediction errors or other language-specific effects. While the representation-level analyses show that SPELL-OUT better preserves trajectory geometry, this does not necessarily demonstrate preservation of semantic progression.

Questions for authors
  1. Regarding the empirical improvements: Are the gains reported in Table 1 consistent across random seeds and statistically significant? Given the relatively small differences compared with Vanilla and SFT + Pres., additional evidence of their robustness would materially strengthen the empirical conclusions.

  2. How much of the observed improvement and representation preservation can be attributed specifically to the relational alignment objective rather than source preservation? In particular, could the authors provide a controlled comparison of these components, including the representation-level analyses?

Reviewer_pzyDOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
3
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes SPELL-OUT, a cross-lingual alignment method for adapting to a target language while preserving capabilities in the source language. It computes residuals between soft token embeddings and reference token embeddings in parallel sentences to align target residuals with source residuals, and also uses a preservation term that keeps the source hidden states close to those of the frozen model before adaptation.

Critical strengths

The motivation to adapt to a target language while maintaining capabilities in existing languages is valid. This is being widely studied, and I think it is one of the practical approaches to addressing the performance degradation that also occurs in the latest on-device or small LLMs. I also think the design of aligning residuals relative to reference tokens is distinctive.

Critical weaknesses
  1. In my view, the current results do not seem to sufficiently demonstrate the effectiveness of the proposed mechanism. The mean target language accuracy increases from 48.0% to 48.2%, while source language accuracy remains the same at 51.9%. Since other adaptation methods reduced performance, this may be a result of damaging the original model less rather than improving performance in the target language. Among papers proposing alignment methods that I have previously seen, some methods improve target language accuracy while almost eliminating the degradation in source language accuracy or even improving source language accuracy as well. This approach could be interpreted as reflecting experimental errors or a misguided research direction. I do not know why this difference occurred, and I think repeated experiments and uncertainty estimation are needed to interpret this small difference. In conclusion, I am not sure what the benefits of your methodology are.

  2. The effect of residual alignment needs to be separated from the effect of small updates. The base training objective has no CE, and when the source residual is small, the method becomes close to regression that reduces the target language residual. Comparing against controls with the source residual set to 0 or shuffled, and against SFT matched in the amount of change, would help determine whether source information actually helps. The error upper bound in the appendix alone does not explain this.

  3. Some related work appears to be missing. Many studies exist, but please consider adding citations to representative work on preserving existing performance in English while aligning to a target language.

  • Cross-Lingual Optimization for Language Transfer in Large Language Models (Lee et al., ACL 2025)
Questions for authors
  • Could you clarify the training data and setup? The main text and Appendix B state that 1,200 Aya examples were used, but Appendix C states that 1,200 Global-MMLU examples were used for training. I would like you to explain the relationship between the actual training data and the evaluation split. The learning rate, preservation layer,weight, and methods for computing the analysis statistics also need to be provided in enough detail to allow reproduction.
  • Overall, the paper seems to use difficult and unnecessary wording, and some figures in particular seem to need improvement. Please consider revising them to communicate more clearly. This seems to be a possible sign of AI drafting. It may make some readers uncomfortable. Of course, as someone who believes that such discomfort should not factor into the evaluation, I did not take it into account in my evaluation score.
Reviewer_aDS3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces SPELL-OUT to adapt multilingual LLMs to target languages without losing source-language performance. It computes token-level prediction residuals in the raw token-embedding table, stretches the source residuals to match the target sequence length using linear position interpolation, and trains the target residuals to match the interpolated source residuals. It also adds an L2 penalty to keep the model's source-language hidden states close to the frozen base model.

Critical strengths
  • Aligning prediction residuals instead of forcing hidden states to be identical across languages is an interesting idea.
  • The evaluation covers five model families and includes representation-drift analyses alongside downstream accuracy.
Critical weaknesses
  • Token-level residual matching without token alignment does not make sense. Matching raw token-embedding residuals by linear position interpolation ignores subword tokenization differences and word-order differences, and assumes raw token embeddings share an aligned coordinate system across languages. The authors mainly test language pairs with very similar grammar, include no ablations against real alignment methods (like Optimal Transport, DTW, or word aligners), and show in Figure 3a that scrambling Subject and Object in half the training data changes accuracy by only 0.138 percentage points.
  • The gains are tiny, baselines degrade performance, and confidence intervals are missing. In Table 1, SPELL-OUT beats the untouched Vanilla model by only 0.2 percentage points (48.2% vs. 48.0%, or 4 questions out of 2,000) and SFT + Pres. by 0.3 percentage points, with zero gain on TinyAya-Global and worse accuracy than SFT on Qwen3-4B (Table 3). No standard deviations are reported, the main baselines (SFT and MLC) perform worse than doing nothing, and stronger continual-learning or cross-lingual baselines cited in the paper are omitted.
  • The source-preservation term is standard activation regularization and drives almost all of the gain. Penalizing L2 drift from the frozen base model's hidden states is standard continual-learning regularization, and Table 1 shows it accounts for 1.5 of the 1.8 percentage point gain over SFT (46.4% to 47.9%). Training scale is too small. Models are trained on only 1,200 examples for 50 to 100 steps (Page 10; Appendix B.1), which is too small to show that the method scales.
  • Manuscript errors and contradictions. Section 2.2 says training uses 1,200 Aya examples, while Appendix C.1 says 1,200 Global-MMLU examples were used for training. Section 2.1 omits the cross-entropy loss term that appears in Section 4.2 and never states which layer(s) use the preservation loss. Appendix C.3 and D are empty headings, several ?? references are broken, and promised Belebele results are missing.
Questions for authors
  1. How does the method perform if you set the source residual to zero, or if you replace linear interpolation with an actual alignment method (such as Optimal Transport, DTW, or word alignment)?
  2. What cross-entropy weight (Section 4.2) was used in Table 1, and which layer(s) were regularized by the preservation loss?
  3. Were the 1,200 training examples sampled from the Aya Dataset (Section 2.2) or Global-MMLU (Appendix C.1)?
  4. How does SPELL-OUT compare against standard source-side KL regularization and the forgetting-mitigation baselines cited in Sections 1 and 5?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.