InfoLat: Informative Latent Communication between Heterogeneous LLMs

The collaboration of heterogeneous large language models (LLMs) is an appealing way to leverage their complementary strengths. Traditional approaches communicate through text, forcing each model to compress its information-rich internal states into discrete tokens. Alternatively, latent communication offers a promising collaboration path by letting models communicate directly in a high-dimensional latent space, without explicit text. However, some existing methods only support communication between models with the same backbone; others learn a communication module for a fixed sender–receiver pair, so replacing the receiver requires retraining from scratch. Moreover, some representative methods tend to collapse the latent message into a task-agnostic one. To facilitate message understanding of heterogeneous receivers, we first represent each latent message as a sequence of latent tokens, where each latent token consists of a set of vocabulary tokens and their continuous weights. Then, we propose \method, a latent communication framework that combines latent label construction, vocabulary alignment, and decoupled sender–receiver training, enabling a sender to broadcast informative latent messages to multiple heterogeneous receivers without retraining the sender. Experiments on math and ALFWorld benchmarks demonstrate that \method consistently outperforms existing communication methods, with superior performance and reduced communication cost.