Semantic Fusion and Subspace Separation in Language Models: A Hypothesis and Its Predictions

We propose a hypothesis of semantic fusion and subspace separation in language models. Early information exchange may build shared representations of semantic units, reducing independent constituent access, while explicit boundaries may favor separately readable subspaces. The hypothesis predicts first-constituent substitutions in predecessor retrieval and their reduction with separators. Across three instruction-tuned language models in English and Chinese, semantic pairs elicit more such errors than random triples, and commas reduce them on paired items. To study the proposed mechanism, we construct a minimal arithmetic-and-retrieval task that permits explicit control over pair meanings and retrieval exposure. A small Transformer exhibits the predicted substitutions for operator pairs assigned arithmetic meanings but withheld from retrieval training. Blocking the early first-to-second-token attention pathway improves retrieval while sharply impairing arithmetic. These findings support the hypothesis's predictions; the proposed subspace organization remains a mechanistic interpretation.