Sparse Context Synthesis: A Formal Framework for Off-the-Shelf LLM Agents

Context optimization improves off-the-shelf large language model (LLM) agents by dynamically injecting context into their prompts rather than fine-tuning model weights. We study a common paradigm of this approach: a catalog of gated factoids, which are small, independently authored functions that inspect an episode and emit short, actionable claims, such as relevant files or policy constraints. We formalize this process as sparse context synthesis: for a given episode, a sparse sub-catalog is activated, and its claims are embedded, weighted on the simplex, and synthesized by the agent into a free-text response. Within this framework, we make three primary contributions. First, we measure how well a catalog fits the evidence by the mutual information its outputs share with a latent target, formalizing three competing design pressures: coverage, redundancy, and localization. Notably, we demonstrate that negative redundancy signals structural synergy among factoids. Second, we establish a theoretical error bridge connecting catalog omissions to downstream agent error, proving that the distance between the embeddings of complete versus omitted claim sets is tightly bounded by their precision and recall deficits. We instantiate this for both observation claims and causal claim graphs to bound end-to-end omission errors. Third, we theoretically ground context importance weights by relating them to expectation-maximization (EM) responsibilities, the Barber–Agakov bound, and information projections, delineating formal proofs from structural analogies. Finally, numerical evaluations on SWE-bench Verified and ${\tau }^2$-bench illustrate the pipeline's operational dynamics, revealing when injected claims successfully bind to guide agent behavior.