Learning Context Compression for LLMs via Implicit Semantic–Structural Decomposition

Context compression enables large language models (LLMs) to reuse long contexts through compact continuous representations, yet existing learning objectives such as reconstruction, continuation prediction, and question answering largely supervise each context in isolation. We revisit context compression from a new semantic–structural perspective, where an effective compressed representation should remain stable under structural changes that preserve meaning, while remaining sensitive to semantic changes under largely preserved structure. Realizing this principle is nontrivial because semantics and linguistic structure are intrinsically entangled and cannot be explicitly separated at the token or representation level. We introduce Semantic–Structural Context Compression (S${}^2$C), which realizes this decomposition implicitly through paired context variants and learning objectives that remain fully compatible with the LLM's native next-token prediction framework. Shared-Target Generation (STG) learns structural invariance by requiring structurally different but semantically equivalent contexts to generate a shared third-view target, thereby suppressing source-specific linguistic form from direct supervision. Response-Shift Matching (RSM) complements this by preserving semantic sensitivity, matching the prediction shifts induced by localized semantic edits under the full context rather than supervising the edited examples independently. Together, STG and RSM separate the effects of structural and semantic variation without explicit latent-space disentanglement or modification of the underlying compression architecture. S${}^2$C consistently outperforms existing context compression methods on six benchmarks, with particularly strong gains at high compression ratios.