Rethinking LLM-based Embeddings: Decompose and Recombine for Better Text Embeddings

Extracting sentence embeddings from large language models (LLMs) has attracted increasing attention without requiring LLM fine-tuning. However, existing methods mainly rely on coarse-grained selection of internal representations. In this work, we decompose hidden states into layer-wise attention and feed-forward network (FFN) components and reveal substantial heterogeneity in their contributions to sentence-level semantics. Based on this observation, we propose DeRe (Decompose and Recombine), which assigns coefficients to individual components and recombines them into the final sentence embedding. Since Spearman correlation is non-differentiable, we employ Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to jointly search the component coefficients based on validation performance. DeRe operates on cached representations without additional LLM forward passes or parameter updates. Extensive experiments on STS, MTEB, and multiple LLM backbones demonstrate consistent improvements across different prompts, tasks, and models.