Equitable Unified Learning for LLM Approximation

Fine-tuning large language models (LLMs) is resource-intensive: it requires a large centralized corpus and substantial compute and memory. This raises three problems. First, such fine-tuning is out of reach for many institutions with limited resources. Second, architecture heterogeneity: institutions choose different architectures suited to their tasks, so models cannot be averaged. Third, data heterogeneity: each institution holds private data that is not independently and identically distributed (non-IID) and must remain local. To address these problems, we propose Unified Learning with Language Models (UniLearnLM), an equitable approach to LLM approximation that fine-tunes small language models (SLMs) through GNN-based unified federated learning. UniLearnLM represents each client's architecture as a typed graph and federates one shared graph neural network (GNN) whose output modulates each frozen backbone through a private adapter, so neither private data nor backbone weights leave a client, client architectures need not match, and under 0.07% of any client's parameters is communicated. Experiments on five language tasks (named entity recognition, question answering, topic classification and two summarization tasks) with four SLM families, and on educational question and answer generation, including a live federation in Tunisia, Germany, Colombia and Peru, show that UniLearnLM is on par with the strongest published heterogeneous federation on single-task federations and yields deeper questions than text-consensus distillation on two testbeds, with depth comparable to local training. Code will be released upon acceptance.