acceptodds
Under review as a conference paper at ICLR 2027

Think Heavy, Act Light: Context-Aware Hierarchical Activation Sparsity for LLMs

Abstract

Activation sparsity has emerged as a highly promising paradigm to alleviate the exorbitant inference costs of Large Language Models (LLMs). However, existing training-free sparsification methods face a fundamental dilemma: they either rely on static, input-agnostic budgets that trigger catastrophic capability collapse on complex reasoning tasks, or employ token-level dynamic routing that incurs prohibitive hardware context-switching overhead and optimization instability. In this paper, we introduce a novel hierarchical dynamic sparsification framework that breaks this deadlock through a planning-and-execution paradigm. Specifically, we elegantly decouple the routing optimization into two stages locked entirely within the compute-bound prefill phase. First, a context-aware Reinforcement Learning (RL) agent acts as a Macro Controller, evaluating prompt complexity to dynamically allocate layer-wise sparsity budgets via a Dense-Guided Policy Optimization strategy. Next, a deterministic Micro Router optimally distributes these assigned budgets across intra-layer modules. By finalizing this prompt-customized configuration prior to generation, our framework degenerates into a static sparse execution graph during the memory-bound autoregressive phase, achieving true zero-overhead decoding. Extensive evaluations across diverse LLM families (e.g., LLaMA and Mistral) demonstrate that our approach comprehensively outperforms existing methods at high sparsity regimes (60%-75%). Notably, while state-of-the-art baselines suffer from severe capability degradation at an extreme 75% sparsity level, our method exhibits graceful degradation, significantly mitigating model output collapse and effectively preserving critical logical reasoning capabilities. Because the sparse execution graph is locked before generation, these accuracy gains come with wall-clock decoding speedups of up to over dense models, matching static sparsifiers at the same sparsity budget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Efficient LLM, Post-training Activation Sparse

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8Hs7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A prompt-conditioned RL controller allocates layer sparsity budgets during prefill, followed by deterministic module allocation and cached magnitude thresholds. Fused kernels apply token-dependent masks during decoding. Evaluation across five backbones reports improved sparse-model perplexity and average task accuracy, with up to 1.71× decoding speedup.

Critical strengths
  1. Moving allocation decisions into prefill addresses a practical source of decoding overhead.
  2. An independently trained prompt-free control isolates prompt conditioning more convincingly than baseline comparisons alone.
Critical weaknesses
  1. The incremental value of RL is insufficiently established. Much of the pipeline overlaps with WAS and related approaches, while the prompt-conditioning control covers one backbone/budget setting. The evidence does not yet establish why RL is preferable to simpler allocation or optimized static scheduling.

  2. Average gains conceal a substantial reasoning tradeoff. Severe GSM8K degradation at high sparsity limits the practical claim of preserving useful model quality. The distribution-matching objective's relationship to reasoning preservation remains unexplained.

Questions for authors
  1. Which capability is new relative to WAS-style budget optimization, module routing, and fused sparse decoding?
  2. Why is RL necessary for prompt-conditioned allocation rather than a simpler conditional predictor or an optimized static schedule?
  3. Does the prompt-conditioning benefit persist across backbones, sparsity levels, and training seeds under matched optimization budgets?
  4. Why does GSM8K degrade so sharply, and what does this reveal about the KL objective's ability to preserve reasoning?
  5. How do realized sparsity and threshold calibration change during long continuations when prefill thresholds remain fixed?
  6. How does the method compare with WAS under a matched decoder and total prefill-plus-decode cost?
7 comments
AuthorsOpenReview ↗

Response to W1 and Q1–Q2: Added capability and the role of RL.

The added capability is learning how to distribute the sparsity budget for each prompt while keeping budget allocation outside the decoding loop. WAS optimizes a shared block-wise schedule; our controller uses the current prompt to choose layer budgets, which the Micro Router converts into module budgets during prefill. The resulting thresholds are then reused during decoding. This gives input-dependent allocation without repeated policy evaluation or budget search for each generated token.

The matched static control directly demonstrates the value of this capability. With the same training data, objective, optimization settings, Micro Router, and decoder, adding prompt information improves nine-task average accuracy from 41.04% to 42.63% on LLaMA-2-7B at 75% sparsity. Appendix E provides complementary evidence on LLaMA-2-13B: reusing budgets from a fixed WikiText prompt reduces BBH and PIQA accuracy by 0.23 and 1.36 percentage points, respectively.

RL provides a direct way to train this conditional predictor from the consequences of its decisions. Layer budgets interact and must satisfy a shared budget constraint, while training data provide no “correct budget” labels. RL evaluates the resulting sparse model through dense-model KL feedback and learns the allocation policy without requiring labeled budget targets or gradients through the discrete sparsification procedure. At inference, the trained policy simply predicts budgets during prefill.

AuthorsOpenReview ↗

Response to W2 and Q4: Reasoning degradation and the role of KL guidance.

The GSM8K results show that reasoning degradation at high sparsity is shared by the evaluated methods. On LLaMA-2-7B at 75% sparsity, GSM8K accuracy falls from 14.03% for the dense model to 0.99% for TEAL, 1.36% for WAS, and 1.36% for ours. This pattern points to a broader challenge of aggressive activation sparsification and does not identify our prompt-conditioned allocation as the cause.

KL guidance helps the sparse model retain the dense model’s token predictions. Its benefit is directly supported by our ablation: removing KL increases WikiText-2 PPL from 11.80 to 13.52 and reduces nine-task average accuracy from 42.63% to 41.54%. Mathematical reasoning requires a sequence of correct steps, however, and an error at a critical step can change the entire continuation. Matching token distributions therefore improves overall prediction quality without ensuring that every reasoning chain reaches the correct answer.

The results establish two distinct findings: KL guidance improves sparse-model quality, while preserving mathematical reasoning at aggressive sparsity remains a shared challenge. Our gains concern overall quality at a given sparsity budget; reasoning-intensive applications require choosing that budget using task-specific accuracy.

AuthorsOpenReview ↗

Response to Q3: Evidence across models, sparsity levels, and training seeds.

The matched prompt-free experiment directly measures the contribution of prompt conditioning. On LLaMA-2-7B at 75% sparsity, both controllers are independently trained with the same data, objective, optimization settings, Micro Router, and decoder:

Macro ControllerWikiText-2 PPL ↓Nine-task average ↑
Prompt-free, learned fixed schedule11.6841.04%
Prompt-conditioned11.8042.63%

Prompt conditioning improves average downstream accuracy by 1.59 percentage points, with a 0.12 increase in PPL. Appendix E adds evidence on LLaMA-2-13B: replacing per-prompt budgets with a fixed WikiText-derived configuration reduces BBH and PIQA accuracy by 0.23 and 1.36 points, respectively.

The broader evaluations establish the applicability of the complete method: it achieves the highest or tied-highest average accuracy among the compared sparse methods across five backbones and all three sparsity levels (60%, 65%, 75%). On the additional Qwen3-8B backbone, average accuracy exceeds TEAL by 0.74 and 1.54 points at 60% and 75%, respectively.

Results on Qwen3-8B.

The table below reports results across nine downstream tasks, with dense performance included as a baseline. All scores are percentages; AVG gives equal weight to each benchmark.

Benchmark     Shots   Dense (0%)   TEAL (60%)   Ours (60%)   TEAL (75%)   Ours (75%)
--------------------------------------------------------------------------------
MMLU              5       76.76        68.96        69.67        54.76        55.92
C-Eval            5       82.24        74.00        74.22        58.92        59.44
ARC-E            25       87.79        86.15        86.70        77.57        79.38
GSM8K             5       82.49        74.98        74.75        36.01        34.72
HellaSwag        10       79.83        72.73        74.55        56.23        59.90
BBH               3       54.38        47.61        48.03        34.93        36.55
WinoGrande        5       76.01        70.09        71.27        58.48        60.14
PIQA              0       79.05        78.35        78.40        75.14        73.56
BoolQ            32       87.71        81.93        83.88        65.50        71.83
--------------------------------------------------------------------------------
AVG                       78.47        72.75        73.50        57.50        59.05
--------------------------------------------------------------------------------

These experiments provide direct evidence for conditioning in the matched setting and complementary evidence across models and budgets. The matched prompt-free comparison uses one training seed; variation across training seeds remains to be measured.

Reviewer_ZMGSOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces "Think Heavy, Act Light," a hierarchical dynamic activation sparsity framework designed to accelerate LLM inference without fine-tuning the backbone. The core innovation lies in decoupling the sparsity routing process into two stages: a "Macro Controller" (a lightweight RL agent) that allocates prompt-conditioned layer-wise sparsity budgets during the prefill phase, and a deterministic "Micro Router" that assigns these budgets to specific modules and caches magnitude-based thresholds. During the autoregressive decoding phase, the model simply uses these cached thresholds to generate masks on the fly, effectively eliminating the routing overhead (e.g., Top-k sorting or policy evaluation) at each decoding step. The authors validate their method on LLaMA-2/3/3.2 and Mistral-7B up to 75% sparsity, demonstrating improved Pareto frontiers over baselines like TEAL and LaRoSA, and reporting up to a 1.71x decoding speedup on a single A800 GPU.

Critical strengths

Elegant Decoupling Paradigm: The proposed "Planning-and-Execution" framework is highly practical. By front-loading the decision-making overhead to the compute-bound prefill stage and caching the thresholds for the memory-bound decoding stage, the authors elegantly address the latency bottleneck that plagues most dynamic routing/sparsity methods.

Empirical Robustness at High Sparsity (for general tasks): Compared to existing post-training sparsity baselines, the proposed method demonstrates graceful degradation on common-sense and reading comprehension tasks, pushing the Pareto efficiency frontier further at extreme sparsity levels (e.g., 75%).

Critical weaknesses

Lack of Compatibility with Continuous Batching (Serving Environments): The empirical speedup (1.71x) is measured strictly at Batch Size = 1. The core contribution of this paper is "prompt-conditioned thresholds," meaning different sequences in a batch will have entirely different sparsity patterns and thresholds at the same layer. It is highly questionable whether this non-uniform thresholding can be efficiently supported by modern batched inference engines (e.g., vLLM, TensorRT-LLM) using batched sparse GEMM without causing severe padding or memory divergence. If the method only works at BS=1, its practical value in real-world deployment is severely limited.

Static Thresholds in Long-Context Generation: Caching thresholds based solely on the prefill activations is a risky "刻舟求剑" (acting without considering changing circumstances) approach for long-context generation. The activation distribution during a short prompt can be vastly different from the distribution hundreds of tokens later (e.g., during a complex Chain-of-Thought process). This static caching mechanism likely contributes to the severe degradation observed in complex reasoning tasks.

Magnitude-only Micro Routing: The Micro Router relies purely on activation magnitude to construct masks. Recent literature has extensively shown that activation magnitude does not equate to importance, and small-magnitude neurons often carry critical, rare features necessary for reasoning and logical deduction.

Questions for authors

Serving / vLLM Compatibility: Can your framework be integrated into continuous batching systems? How do you propose to handle batched matrix multiplications where each sequence in the batch requires a different, prompt-specific magnitude threshold? Please discuss this implementation bottleneck.

Long-context & Reasoning Collapse: The performance on GSM8K drops drastically (e.g., 1.52 on LLaMA-3-8B at 75%). Furthermore, there is no evaluation of coding tasks (e.g., HumanEval, MBPP), which are similarly sensitive to logical disruption. Could you provide code generation metrics? I suspect the static prefill-based thresholding and magnitude-only routing destroy the long-term logical chains.

Discussion on Reasoning Collapse Literature: Please contextualize your severe math degradation with recent findings in the field. Specifically, works like "Rethinking the value of training-free structured pruning of LLMs" point out that training-free sparsity creates a false sense of security on simple tasks (like HellaSwag) while collapsing on reasoning. Additionally, "Reasoning-preserved Efficient Distillation of Large Language Models via Activation-aware Initialization" highlights that even models optimized through compression and distillation face severe "reasoning collapse", a phenomenon closely related to the drop in the effective rank of hidden states. How does your magnitude-only masking method impact this effective rank, and how might it address or suffer from these fundamental issues?

Alternative Metric Strategies: Instead of relying solely on magnitude, could the Micro Router accommodate different metric strategies (e.g., Taylor expansion, activation-weight norm scaling)? Have you experimented with alternative metrics to rescue the math/coding performance?

Dynamic Width vs. Dynamic Depth: How does your dynamic activation sparsity (Dynamic Width) compare in terms of wall-clock speedup and capability retention with dynamic layer routing methods (Dynamic Depth), such as "Dr.LLM: Dynamic Layer Routing in LLMs"? Skipping an entire layer saves memory bandwidth directly, which might be more efficient than masking neurons while still loading the weights.

Performance on Instruct / Strong Reasoning Models: The evaluations are exclusively on base models. How robust is your "Think Heavy, Act Light" approach on these strong instruction-tuned models (Qwen-Instruct)? Does the reasoning capability degrade even faster?

8 comments
AuthorsOpenReview ↗

Response to Q1: Continuous batching and deployment relevance

We agree that continuous batching is essential for cloud serving and represents an important future extension of our framework. Meanwhile, batch-size-1 decoding has clear practical relevance: beyond single-user applications on mobile and edge devices, developers also deploy LLMs locally on personal GPUs for private, interactive use. These latency-sensitive workloads may have only one active request and therefore cannot rely on cross-request batching.

Our reported 1.71× speedup demonstrates the benefit of single-request decoding on an A800 and excludes prefill-time planning and threshold construction. The scenarios above motivate the practical relevance of this evaluation setting. Prompt-specific thresholds do not inherently prevent batching, since each request can retain its own cached thresholds. The main challenge is executing different active-channel patterns across requests efficiently while preserving weight reuse and memory efficiency. This depends substantially on kernel-level optimization, while our algorithmic framework primarily focuses on preserving the accuracy of the sparsified model.

AuthorsOpenReview ↗

Response to Q2: Static thresholds and reasoning degradation.

Our method fixes thresholds, not masks: each decoding step applies the cached thresholds to the current activations, so active neurons remain token dependent. This retains dynamic selection while avoiding repeated policy evaluation, routing, and percentile computation.

Reasoning degradation at high sparsity also occurs in existing methods. On Qwen3-8B, GSM8K accuracy drops from 82.49% for the dense model to 36.01% with TEAL and 34.72% with ours at 75% sparsity. Thus, this degradation cannot by itself establish our prompt-cached thresholds as the cause. Meanwhile, our method improves nine-task average accuracy over TEAL by 1.54 percentage points at this sparsity, demonstrating better overall capability retention, although not uniformly across tasks.

We acknowledge that prolonged generation can introduce distribution drift; dynamic masks do not eliminate this risk. Evaluating threshold refresh against its additional overhead is an important extension.

AuthorsOpenReview ↗

Response to Q3: Micro Router and feature importance

Our Micro Router does not allocate sparsity solely according to activation magnitude. Subject to the layer budget, it greedily assigns additional sparsity to the module that yields the smallest marginal increase in dense–sparse layer-output reconstruction MSE. This allocation therefore accounts for the effects of sparsification on the output, including the influence of module weights. The Macro Controller further optimizes layer budgets using dense-guided KL feedback.

Magnitude thresholding is the final masking rule and is also used by TEAL; its limitations are not unique to our framework. Our contribution is to combine this inexpensive rule with prompt-conditioned budgeting and output-error-aware module allocation, rather than treating magnitude as a complete measure of importance. MSE does not guarantee preservation of every rare reasoning feature, but characterizing the entire router as “magnitude-only” overlooks its explicit reconstruction-based allocation mechanism.

Reviewer_Z3P4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

To address the performance-overhead dilemma inherent in conventional training-free activation sparsity, the authors introduce a novel hierarchical dynamic sparsification framework which adopts planning-and-execution paradigm. During the prefill stage, an RL-based Macro Controller evaluates prompt complexity and allocates layer-wise sparsity budgets via Dense-Guided Policy Optimization. Subsequently, Micro Router distributes these assigned budgets across intra-layer modules, finalizing a prompt-tailored configuration. In the decoding phase, it constructs a static sparse execution graph, achieving true zero-overhead decoding. In reported experiments, the authors' method outperforms existing approaches, matching static sparsifiers under the same sparsity budget.

Critical strengths

This paper addresses a clear and crucial tradeoff between performance and overhead in sparsification frameworks. The approach is reasonable to achieve performance on par with static sparsification (zero-overhead decoding).

Critical weaknesses

Method novelty: The overall mechanism is remarkably similar to WAS, with differences limited to minor implementation details (e.g., which algorithm determines the budget, what serves as the threshold). As a result, I feel that the target problem is too narrowly scoped relative to the general theme of sparsification frameworks. Can’t find empirical analysis for the underlying premise: layer sensitivity varies across prompts. While Eq. 2 claims to minimize task loss, it actually employs KL divergence in practice without clarifying the connection between the two. Missing necessary ablations: The paper does not ablate which component (Micro Router, the prompt-specific prefill threshold, or the RL-based layer budget allocation) drives the performance difference . Furthermore, the algorithm (e.g., PPO) and the reward formulation should be ablated separately. The experimental validation is thin. The main text lacks an analysis of how much the learned policy actually varies budgets across prompts (e.g., variance, cross-task differences). Generalizability needs to be demonstrated through experiments on other models beyond LLaMA and Mistral, across diverse architectures and model sizes. Based on the results, the largest improvements observed under the 75% sparsity regime is a practically uncommon condition. But this is not discussed in the paper.

Questions for authors

See the above weakness. Could you evaluate performance across a wider range of sparsity ratios and plot the trend curves of performance and overhead as a function of sparsity (x-axis)?

6 comments
AuthorsOpenReview ↗

Response to W1: Input-dependent layer sensitivity.

Table 6: Post-hoc cross-task schedule transfer on LLaMA-2-13B at 75% sparsity.

“Native” uses the allocation predicted for each prompt. Fixed schedules are task-level means derived from the trained conditional policy, not independently optimized prompt-free controls. The fixed WikiText schedule is computed from a single WikiText prompt. Higher is better.

Allocation used at evaluation      BBH (3-shot)  PIQA (0-shot)  Average
----------------------------------------------------------------------
Native prompt-conditioned                30.64          75.30    52.97
Fixed PIQA-derived schedule              30.06          74.81    52.44
Fixed BBH-derived schedule               30.47          73.07    51.77
Fixed WikiText prompt schedule           30.41          73.94    52.18
----------------------------------------------------------------------

Cross-task transfer of fixed allocations.

We investigate task-dependent allocation preferences through a controlled cross-task transfer study on LLaMA-2-13B at 75% sparsity, with details provided in Appendix E.

On PIQA, the PIQA-derived fixed schedule achieves 74.81% accuracy, whereas the BBH-derived schedule achieves 73.07%, a decrease of 1.74 percentage points. On BBH, the BBH-derived schedule achieves 30.47% accuracy, compared with 30.06% for the PIQA-derived schedule. Per-prompt allocation achieves the highest accuracy among these configurations on both benchmarks: 75.30% on PIQA and 30.64% on BBH.

Fixed allocation from a single WikiText prompt.

We additionally compute a budget configuration from a single WikiText prompt and reuse it across both benchmarks. As reported in Table 6, this fixed configuration achieves 30.41% accuracy on BBH and 73.94% on PIQA, falling below our prompt-adaptive method by 0.23 and 1.36 percentage points, respectively. The exact WikiText prompt is provided in Appendix E for reproducibility.

Interpretation.

Under the same global sparsity budget, task-derived schedules exhibit different cross-task transfer performance, and per-prompt adaptation improves accuracy over the evaluated fixed schedules. These results provide empirical support for prompt-conditioned budget allocation. They constitute indirect evidence consistent with input-dependent layer sensitivity, rather than direct measurements of individual layers' sensitivity.

AuthorsOpenReview ↗

Response to W2: ablation study

1. Component contributions and the role of the KL objective

Thank you for requesting a clearer justification and more targeted ablations. We agree that the original experiments did not fully isolate the contributions of the Micro Router, thresholds computed from the current prompt's prefill, and RL-based layer-budget allocation.

Table 3 provides a matched comparison specifically for prompt conditioning in the Macro Controller. The prompt-free and prompt-conditioned policies are trained independently using the same data, dense-guided PPO objective, optimization settings, Micro Router, target sparsity budget, and fused decoder. They differ only in whether the Macro Controller receives prompt-dependent features. Here, “prompt-free” applies only to the Macro Controller; the activation-dependent Micro Router remains unchanged.

On LLaMA-2-7B at 75% sparsity, prompt conditioning improves nine-task average accuracy from 41.04% to 42.63%, a gain of 1.59 percentage points, while WikiText-2 perplexity increases slightly from 11.68 to 11.80. This supports the contribution of prompt conditioning to average downstream accuracy in this setting, rather than an improvement in every metric. This comparison does not independently isolate the Micro Router or the prefill-based threshold mechanism.

We have also clarified the discussion around Eq. 2 and the KL reward to distinguish the task-level objective from the distillation surrogate used for policy optimization. For a fixed input and a frozen dense teacher,

= \mathbb{E}_{y \sim P_d}[-\log P_s(y)] - H(P_d).$$ Because the teacher entropy is independent of the sparsity policy, minimizing this KL divergence is equivalent to minimizing cross-entropy against the teacher's soft targets. This encourages preservation of the dense model's predictions under the sparsity budget without requiring downstream-task labels. It is a surrogate objective, not an equivalence to every downstream loss; preservation of task performance is evaluated empirically through downstream accuracy and perplexity. **2. Separating reward design from the optimization algorithm** To isolate the contribution of KL guidance while holding the optimization algorithm fixed, we add a prompt-conditioned PPO ablation that removes the KL reward term. The updated results are summarized below. **Table 3. Ablation results on LLaMA-2-7B at 75% sparsity.** PPL is measured on WikiText-2 (lower is better); Avg. is nine-task average accuracy in percent (higher is better). ```text Policy variant PPL Avg. (%) ------------------------------------------------------------ Naive policy gradient 13.59 40.45 Dense-guided PPO, prompt-free 11.68 41.04 Dense-guided PPO, prompt-conditioned 11.80 42.63 Prompt-conditioned PPO, without KL 13.52 41.54 ------------------------------------------------------------ ``` Removing KL guidance while retaining prompt conditioning and PPO increases perplexity from 11.80 to 13.52 and reduces average accuracy from 42.63% to 41.54%, a decrease of 1.09 percentage points. Together with the prompt-free comparison, this provides evidence for distinct contributions from prompt conditioning and KL guidance under the evaluated configuration.
AuthorsOpenReview ↗

Response to W3: Evaluation beyond LLaMA and Mistral

Thank you for raising this concern. We have added Qwen3-8B experiments, together with the corresponding analysis and conclusions, to Appendix H of the revised manuscript. This extension evaluates our method on an additional model family at target sparsities of 60% and 75%.

Experimental setup.

Following the training settings in Section 4.1, we train a separate prompt-conditioned PPO/KL policy for each target on WikiText-103, with frozen backbone weights and no downstream-task supervision. These experiments evaluate the applicability of our framework to an additional backbone with retraining, rather than direct transfer of an existing LLaMA or Mistral policy.

Results on Qwen3-8B.

The table below reports results across nine downstream tasks, with dense performance included as a baseline. All scores are percentages; AVG gives equal weight to each benchmark.

Benchmark     Shots   Dense (0%)   TEAL (60%)   Ours (60%)   TEAL (75%)   Ours (75%)
--------------------------------------------------------------------------------
MMLU              5       76.76        68.96        69.67        54.76        55.92
C-Eval            5       82.24        74.00        74.22        58.92        59.44
ARC-E            25       87.79        86.15        86.70        77.57        79.38
GSM8K             5       82.49        74.98        74.75        36.01        34.72
HellaSwag        10       79.83        72.73        74.55        56.23        59.90
BBH               3       54.38        47.61        48.03        34.93        36.55
WinoGrande        5       76.01        70.09        71.27        58.48        60.14
PIQA              0       79.05        78.35        78.40        75.14        73.56
BoolQ            32       87.71        81.93        83.88        65.50        71.83
--------------------------------------------------------------------------------
AVG                       78.47        72.75        73.50        57.50        59.05
--------------------------------------------------------------------------------

At the 60% target sparsity, our method improves average accuracy from 72.75% to 73.50%; at the 75% target, it improves average accuracy from 57.50% to 59.05%. The gains, computed before rounding, are 0.74 and 1.54 percentage points, respectively. Our method outperforms TEAL on eight of nine tasks at 60% and seven of nine tasks at 75%.

At the 75% target, the largest gains occur on BoolQ (+6.33 percentage points) and HellaSwag (+3.66), with additional improvements on ARC-E, WinoGrande, BBH, MMLU, and C-Eval. Performance decreases on GSM8K (-1.29 percentage points) and PIQA (-1.58). Thus, the average improvement does not imply a gain on every task. Understanding and mitigating these task-specific regressions remains a direction for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.