acceptodds
Withdrawn from ICLR 2027

Multilingual Latent Representation Distortion Drives Generative Drift and Bounds Latent-Channel Communication Capacity

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multilingual large language models (LLMs) internalize conceptual reasoning within an English-centric latent frame, while low-resource languages (LRLs) occupy activation subspaces that are compact relative to English ( under mean pooling; – per token excluding the BOS position). Across seven LLM backbones, Belebele configurations and FLORES+ varieties, we show that this single geometric property is closely associated with two failures previously studied in isolation: (i) intra-model language drift during free-form generation, and (ii) the limited capacity of continuous latent channels for inter-agent communication. First, contrastive Singular Value Decomposition isolates a low-rank language surface subspace from an orthogonal reasoning subspace that is cross-lingually isomorphic to English in a model-invariant way (debiased CKA, Procrustes, and RSA; Spearman's - across six model lineages). Building on this geometry, we design a training-free, depth-stratified and magnitude-normalized steering method that bounds relative layer-wise perturbation. Our method reduces Involuntary Fidelity Loss by on low-resource scripts (Macro IFL on SEA-LION-v3-8B) and by - relative across six backbones, while leaving safety-classification accuracy within confidence intervals. Second, we introduce a specificity protocol for continuous multi-agent latent channels across nine languages, and a Universal Latent Hub that extends continuous communication to heterogeneous model pools with mismatched hidden dimensions. We establish the token-budget crossover where continuous latents outperform literal text on high-fertility scripts. Finally, we examine the unifying bridge: on the six languages shared by both programs, activation distortion and tokenizer fertility are collinear ( under mean pooling), suggesting a single resource-scarcity axis associated with both single-model generative drift and multi-agent channel capacity, a pattern consistent with fertility and accuracy trends across Belebele configurations.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
multilingual LLMs, cross-lingual alignment, low-resource languages, representation engineering, activation steering, continuous latent communication, multi-agent systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7LemOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multilingual representation geometry and its connection to language drift and latent communication. The authors use contrastive SVD to separate language-specific directions from a shared reasoning subspace, and propose training-free steering with depth scheduling and magnitude normalization to reduce drift. They also introduce a latent hub connecting models with different hidden dimensions and controls that distinguish content transfer from fixed prefix cues. Experiments across models and languages examine these methods and relate representation distortion to tokenization and communication performance.

Critical strengths
  • Meaningful research scope: The paper studies multilingual representations in both single-model generation and multi-agent communication. It examines their connection to language drift and information transfer. I think the controlled prefix experiments are useful for distinguishing instance-specific content transfer from input-independent cues. These controls help clarify whether apparent communication gains reflect actual information transfer.

  • Broad empirical coverage: The evaluation covers languages with varying resource levels and multiple model families. It also uses complementary geometric and behavioral metrics. This breadth helps assess whether the observed patterns extend beyond a particular language, model, or measurement.

Critical weaknesses
  • Validation could be more complete: The geometric analysis shows associations, but provides limited evidence that distortion explains both failures. The steering results also leave some uncertainty, as the full method reduces drift but hurts accuracy on some models, while the gated variant preserves accuracy but lacks corresponding drift results in Table 2. A more direct evaluation of these claims would strengthen the paper.

  • The two research directions could be better connected: The steering and latent communication components are interesting, but their connection is mainly supported by cross-language correlations. Testing whether the proposed subspaces affect communication would help clarify the value of the unified geometric explanation.

Questions for authors
  • Effect of pretraining language distribution: Could the observed geometry depend on the backbone’s pretraining language mix? For example, Qwen may have greater exposure to Chinese than some other backbones. Would using Chinese or another well-supported language as the reference change the distortion patterns or steering results?

  • Interpretation of the reasoning subspace: What evidence distinguishes reasoning-specific functionality in URU_R from broader language-agnostic semantic processing? The alignment and drift experiments seem compatible with both interpretations. Are there targeted interventions showing that this subspace specifically supports reasoning?

  • Evaluation of baselines and components: Could the authors report drift and task accuracy together for the gated variant and tuned steering baselines? It would also help to evaluate the G/M/P ablations on free-generation drift, since the current factorial measures teacher-forced accuracy. This would clarify which components drive drift reduction and what performance trade-offs they introduce.

I would be willing to raise my score if the authors could clarify these points and provide stronger evidence for the main claims.

Reviewer_tVCUOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that inside multilingual LLMs, low-resource languages have smaller activation norms than English, and that this explains two problems: generations drifting out of the target script into English, and the limited capacity of continuous latent channels between agents.

For drift, it proposes training-free steering. It reports lower script-based Involuntary Fidelity Loss on six languages (0.491 → 0.295 on SEA-LION-v3-8B) and compares Belebele accuracy against steering baselines.

For communication, it trains a soft-prefix projector between two copies of Qwen2.5-7B on a verbatim-reproduction task and compares matched, mismatched, constant, no-prefix and truncated-text conditions across nine languages. It links the two parts through correlations between distortion, tokenizer fertility and channel specificity over six languages.

Critical strengths

None

Critical weaknesses
  • The paper is very hard to read and contains errors that show it was not checked:
    • Many terms are branded or undefined: "CLAS" (Figure 6) and "H2 Control" (Table 7) are never defined, the "Four-Arm" protocol has five arms (L270), LPR is defined two different ways (L309, L352), the captions of Figures 3(b) and 5 describe things the plots do not show, and the caption of Table 11 contains file paths (L936, L939).
    • Nine references have wrong authors, titles or arXiv IDs (listed in the ethics field). For example, the arXiv ID given for AfriMGSM (2311.08358) belongs to an astrophysics paper.
    • The LLM usage statement says all claims and results were "reviewed and verified" by the authors (L627–629), which these errors contradict.

This matters because the claims cannot be checked against the text, and the errors indicate the manuscript was not verified before submission.

Questions for authors

None

Reviewer_cfpkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates language drift and poor latent communication in multilingual LLMs, proposing that both arise from distorted representations of low-resource languages. It introduces an activation-steering method to reduce drift and a controlled protocol to distinguish message-specific information transfer from generic prefix effects.

Critical strengths

Language-drift reduction is evaluated across 6 model backbones. I also appreciate the transparent reporting of negative downstream communication results and limitations of the proposed explanation.

Critical weaknesses

I appreciate the authors reporting the negative downstream communication results. Given these results, an important contribution of the work is its proposed explanation of language drift in low-resource languages and poor latent communication. But, the evidence connecting these problems is limited. The key analysis uses 6 languages only and the correlation between representation distortion and communication specificity is not statistically significant (-0.59). It further weakens after controlling for fertility (-0.19). This does not establish that representation distortion is a common cause of communication failure and language drift. Tokenization, training data coverage, or general language competence could also explain the observed patterns.

The interpretation of the remaining dimensions as a reasoning subspace can be validated more. SVD identifies dominant differences between parallel English and target-language representations, but removing those differences does not establish that the remaining dimensions are language-independent.

Questions for authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.