Think Heavy, Act Light: Context-Aware Hierarchical Activation Sparsity for LLMs

Activation sparsity has emerged as a highly promising paradigm to alleviate the exorbitant inference costs of Large Language Models (LLMs). However, existing training-free sparsification methods face a fundamental dilemma: they either rely on static, input-agnostic budgets that trigger catastrophic capability collapse on complex reasoning tasks, or employ token-level dynamic routing that incurs prohibitive hardware context-switching overhead and optimization instability. In this paper, we introduce a novel hierarchical dynamic sparsification framework that breaks this deadlock through a planning-and-execution paradigm. Specifically, we elegantly decouple the routing optimization into two stages locked entirely within the compute-bound prefill phase. First, a context-aware Reinforcement Learning (RL) agent acts as a Macro Controller, evaluating prompt complexity to dynamically allocate layer-wise sparsity budgets via a Dense-Guided Policy Optimization strategy. Next, a deterministic Micro Router optimally distributes these assigned budgets across intra-layer modules. By finalizing this prompt-customized configuration prior to generation, our framework degenerates into a static sparse execution graph during the memory-bound autoregressive phase, achieving true zero-overhead decoding. Extensive evaluations across diverse LLM families (e.g., LLaMA and Mistral) demonstrate that our approach comprehensively outperforms existing methods at high sparsity regimes (60%-75%). Notably, while state-of-the-art baselines suffer from severe capability degradation at an extreme 75% sparsity level, our method exhibits graceful degradation, significantly mitigating model output collapse and effectively preserving critical logical reasoning capabilities. Because the sparse execution graph is locked before generation, these accuracy gains come with wall-clock decoding speedups of up to $1.71\times$ over dense models, matching static sparsifiers at the same sparsity budget.