acceptodds
Withdrawn from ICLR 2027

Neat Cross-Domain Representations via Frozen Embedding-Anchored Training of Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Traditional language model training tunes all components jointly to converge on a suitable representation. Can we align separately trained models on a shared embedding space? We use pre-trained and frozen embeddings to initialize auto-regressive LMs and anchor the representation space throughout (randomly initialized) backbone training. We focus on very small model scales (< 100M parameters). We pre-train multiple models independently on different domain datasets and show that representation alignment emerges consistently on models trained within our simple anchor setting, as measured by centered cosine distance and CKA over representations of identical contexts between different models. Despite this property, each model performs better in its respective domain.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
language modeling, representation learning, transfer learning

Related papers

Discussion (0)

Sign in to comment.