Neat Cross-Domain Representations via Frozen Embedding-Anchored Training of Language Models

Traditional language model training tunes all components jointly to converge on a suitable representation. Can we align separately trained models on a shared embedding space? We use pre-trained and frozen embeddings to initialize auto-regressive LMs and anchor the representation space throughout (randomly initialized) backbone training. We focus on very small model scales (< 100M parameters). We pre-train multiple models independently on different domain datasets and show that representation alignment emerges consistently on models trained within our simple anchor setting, as measured by centered cosine distance and CKA over representations of identical contexts between different models. Despite this property, each model performs better in its respective domain.