acceptodds
Under review as a conference paper at ICLR 2027

When Is Component Sensitivity Compositional? A Causal Study of Neural Resource Allocation

Abstract

Neural resource-allocation methods commonly assign each component a scalar sensitivity and rank components independently. This raises two distinct questions: whether the score tracks singleton utility, and whether ranking by singleton utility remains useful under joint allocation. We separate these two questions and test them through direct interventions under a common, logged protocol across quantization, layer skipping, structured pruning, mixture-of-experts selection, and adapter placement. The resulting behavior forms a spectrum. Layer skipping exhibits small median pairwise interactions, yet one-shot singleton selection can still incur substantial held-out NLL loss at the allocation budget; pruning rankings depend on background sparsity; GPTQ-based group refinement and adapter placement show strong non-additivity; and expert interactions vary across the tested depths. Large additive prediction errors need not imply large selection gaps: the six-layer adapter placement realizes only 15% of its singleton-sum NLL prediction yet trails the tested last-6 heuristic by only 0.011 nats. By contrast, on Qwen2.5-14B one-shot singleton selection at a 25% layer-skip budget incurs 1.013 nats of held-out damage versus 0.540 for sequential re-measurement, and under sparse pruning backgrounds the dense ranking transfers worse than background re-measurement in 16 of 18 conditions—separating additive prediction failure from realized allocation loss. The 14B layer-skip NLL advantage does not yield a statistically resolved accuracy advantage on ARC-Challenge or HellaSwag. As an exploratory diagnostic, residual-stream perturbation alignment is positively associated with pairwise interaction magnitude in 11 of 12 runs. We conclude that component sensitivity need not behave as an intrinsic scalar: it is a configuration-dependent marginal effect in several tested regimes, and allocation methods should validate compositionality at the deployment granularity, background, budget, and metric.

Primary area:
interpretability and explainable AI
Keywords:
Component Sensitivity, Compositionality, Neural Resource Allocation, Model Compression, Causal Intervention, Neural Network Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HwkiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether component sensitivities measured through individual interventions remain useful for joint resource allocation. It distinguishes proxy fidelity, additive prediction error, and realized selection loss across quantization, layer skipping, structured pruning, MoE expert selection, and adapter placement. The experiments show that sensitivity depends on the background configuration, allocation budget, and evaluation metric. The paper also explores residual-stream perturbation alignment as a correlate of pairwise interactions.

Critical strengths
  1. Useful conceptual distinctions. The paper clearly separates inaccurate additive predictions from poor allocation decisions. The observations that large prediction errors can coexist with small selection gaps, and small pairwise interactions with substantial budget-scale losses, are informative.

  2. Thoughtful experimental controls. Coherent quantization interventions, a compensation-free quantization control, frozen-assignment MoE experiments, and matched-removal-count comparisons help address plausible alternative explanations.

  3. Transparent validation and reporting. The study includes held-out evaluations, repeated seeds, selected cross-corpus checks, and downstream accuracy audits. It explicitly reports attenuated effects and cases where NLL improvements do not translate into statistically resolved accuracy gains.

Critical weaknesses
  1. Limited advance over established observations. As acknowledged in Section 9, prior methods such as SLEB already account for importance changing after other components are removed. The contribution is primarily a broader empirical characterization; the paper does not sufficiently establish a new generalizable explanation beyond configuration-dependent importance and component interactions.

  2. The diagnostics remain descriptive. The framework characterizes observed failures but does not establish when singleton ranking will be reliable in an unseen configuration. Some operational criteria already incorporate measured greedy-selection gains, while residual-stream alignment remains an exploratory correlate without validation as a predictor of allocation loss.

  3. Practical allocation implications are limited. Strong non-additivity does not consistently produce substantial selection penalties. For example, the 1.5B adapter experiment reports a large singleton-sum overprediction but only a 0.011-nat gap to the last-six-layer heuristic. The strongest layer-skipping NLL advantage also yields no statistically resolved improvement on the two evaluated downstream tasks. These results support metric-specific conclusions more strongly than broad claims about practical allocation quality.

Questions for authors
  1. Which finding provides the strongest advance over existing work on configuration-dependent importance and iterative re-scoring, such as SLEB? What evidence supports its generality beyond the tested configurations? 2.Have any proposed diagnostics been evaluated for predicting held-out allocation loss across configurations or budgets, without using the observed selector advantage in the diagnostic itself? Please report existing results, if available. 3.What evidence supports the stability of the pruning conclusions under larger independent evaluation samples or less degraded backgrounds? Were compensated or recovered sparse models examined, and do any available results preserve the reported ranking changes and selection penalties?
Reviewer_6RxMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines whether component-level sensitivity scores are reliable for joint resource allocation in LLMs. It studies quantization, layer skipping, structured pruning, MoE expert selection, and LoRA adapter placement using a common intervention-based framework. The results show that component importance can change with the model configuration, and even directly measured individual sensitivities can lead to poor joint selections.

Critical strengths
  • The paper clearly separates errors in proxy estimation, additive prediction, and actual selection decisions.

  • The experiments cover five resource allocation settings and models of different sizes, giving a useful view of how compositionality varies across tasks.

  • The intervention protocols are carefully designed, with several useful control experiments. The comparison between static ranking and sequential re-measurement also shows substantial NLL selection gaps at certain layer-skipping budgets.

Critical weaknesses
  • The primary sensitivity measurements in the forward-only experiments use eight 512-token sequences. Although Appendix D includes additional C4 evaluations, some pruning effects become substantially weaker (Table A33). It remains unclear whether the rankings and interaction estimates in other settings would hold with larger and more diverse evaluation sets.
  • Table A30 reports the measurement costs of different selectors, but the quality–cost trade-off is not explored in much detail. It would be useful to see whether limited re-measurement can recover most of the gains from full sequential greedy selection at a lower search cost.
  • The selection experiments vary considerably across domains. Sequential greedy selection is only tested for layer skipping, while the exact-singleton quantization test covers one layer with 12 candidate blocks (Tables A5, A23, and A30). More selection-level comparisons would help clarify whether adaptive selection offers similar benefits in the other settings.
Questions for authors

Sequential greedy selection is only evaluated for layer skipping. Would similar improvements be expected in pruning, quantization, or MoE expert selection? Are there any preliminary results supporting this?

Reviewer_w5yHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the compositionality of component sensitivity in neural resource allocation. The authors conduct studies across five domains (layer skipping, structured pruning, quantization, MoE routing, and adapter placement) to test whether the effect of dropping multiple components equals the sum of their individual effects. They find a broad spectrum of behavior: quantization and adapter placement are highly non-compositional, structured pruning rankings fail to transfer to sparse backgrounds, and layer skipping shows small pairwise interactions but still suffers from allocation loss at budget scales. They also explore a residual-stream coupling metric to predict these interactions.

Critical strengths
  1. I think the paper does a good job of separating the failure of the scoring metric (proxy fidelity) from the failure of the components to act independently (additive prediction error). By measuring the exact singleton utility, the authors show that even if we perfectly know the effect of removing one component, adding up those individual effects still does not predict what happens well when you act on several at once, in domains like quantization and adapter placement.

  2. I appreciate the practical distinction between a prediction failing and the actual model getting worse. For example, the authors show that the 7B adapter placement has massive additive prediction error, but the actual realized allocation loss compared to a simple last-6 heuristic is very small.

Critical weaknesses
  1. Nearly every effect is measured in held-out NLL, but in the two places the authors check a deployed metric, the NLL effect does not survive. The 14B layer-skip greedy advantage of +0.47+0.47 nats [+0.25,+0.75][+0.25, +0.75] yields no resolved accuracy difference on ARC-Challenge or HellaSwag (−0.3-0.3 pp, p=0.90p=0.90; +0.6+0.6 pp, p=0.07p=0.07), and no robust-EM contrast among adapter placements, nor against the base at 7B, is statistically resolved. So the "allocation losses" that motivate the entire study are demonstrated almost exclusively in a surrogate that, wherever it is actually tested against task performance, does not move. This is the single consideration that most holds back my recommendation: the paper measures a real phenomenon very carefully, but has not shown that the phenomenon matters for what practitioners deploy. (I separate this from the honesty of reporting it, which I count as a strength.)

  2. As the authors note in the limitations, the scale coverage is uneven (layer skipping up to 32B, pruning up to 14B, adapters up to 7B). Because width, depth, and pretraining change across these models, I think it is difficult to confidently isolate whether a behavior belongs strictly to the intervention type (like adapter placement) or is just an effect of the specific model scale and training recipe.

Questions for authors
  1. If residual-stream coupling is only an exploratory correlate, what concrete algorithmic changes do you propose to actually solve the quantization or adapter placement problems you identified?
  2. How does your diagnostic framework inform the choice between one-shot allocation and established iterative pruning baselines in production environments?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.